2024-09-15T20:15:00.693 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-15T20:15:00.714 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T20:15:00.795 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600 branch: main description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_api_tests} email: ceph-qa@ceph.com first_in_suite: false job_id: '7905600' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 global: debug ms: 1 debug objecter: 20 debug rados: 20 mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 25 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub inject crc mismatch: 0.01 mon scrub inject missing keys: 0.05 mon scrub interval: 300 mon warn on pool no app: false paxos min: 10 paxos service trim min: 5 paxos trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore write v2: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - ScrubResult - scrub mismatch - overall HEALTH_ - \(MGR_DOWN\) - \(PG_AVAILABILITY\) - \(SLOW_OPS\) - slow request - \(OSD_SLOW_PING_TIME - overall HEALTH_ - \(MON_DOWN\) - \(POOL_APP_NOT_ENABLED\) - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_FULL\) - \(SLOW_OPS\) - \(MON_DOWN\) - \(PG_ - \(SMALLER_PGP_NUM\) - slow request mon_bind_msgr2: false sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 7498493279982dfe87f00616198a5967475f6169 owner: scheduled_teuthology@teuthology priority: 101 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - osd.0 - osd.1 - osd.2 - - mon.b - mgr.x - osd.3 - osd.4 - osd.5 - client.0 seed: 2855 sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 sleep_before_teardown: 0 subset: 17936/100000 suite: rados suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 7498493279982dfe87f00616198a5967475f6169 targets: smithi078.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDeyI8ErHjmywyAcgaaMxyLBjW6vQ/ZTkX96c5IAX6jwub0/UwN3ibsTL6P/F4ucTPWzc0diLCQKX1b5ouizXwA= smithi146.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXaqSChWCsjJib9s1kXCCHha6GFby9mTWPWasr8GTz3rx/YfyXumXWeqhTzNH4vaqHa7qSVZxuF7//bK3bqR6g= tasks: - install: null - ceph: null - mon_thrash: revive_delay: 90 thrash_delay: 1 - workunit: clients: client.0: - rados/test.sh unit_test_scan teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-15_20:00:16 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-15T20:15:00.795 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa; will attempt to use it 2024-09-15T20:15:00.796 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa/tasks 2024-09-15T20:15:00.796 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-15T20:15:00.798 INFO:teuthology.task.internal:Checking packages... 2024-09-15T20:15:00.819 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'fc3e0bff7468fd55478adbae7f75508b749124b6' 2024-09-15T20:15:00.819 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-15T20:15:00.819 INFO:teuthology.packaging:ref: None 2024-09-15T20:15:00.819 INFO:teuthology.packaging:tag: None 2024-09-15T20:15:00.819 INFO:teuthology.packaging:branch: main 2024-09-15T20:15:00.819 INFO:teuthology.packaging:sha1: fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:15:00.819 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=main 2024-09-15T20:15:01.043 INFO:teuthology.task.internal:Found packages for ceph version 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:15:01.045 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-15T20:15:01.056 INFO:teuthology.task.internal:no buildpackages task found 2024-09-15T20:15:01.056 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-15T20:15:01.074 INFO:teuthology.task.internal:Saving configuration 2024-09-15T20:15:01.088 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-15T20:15:01.097 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-15T20:15:01.122 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi078.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 20:03:55.139325', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII3vcWIf1swLneW9Qu7nWoan17XLQ3CcX9o1N7BQH8zP'} 2024-09-15T20:15:01.143 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi146.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-15 20:03:55.137932', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ0TxuRLzADhv5ci6agnm9+lSJjtdgaE9Mv2AUuEF7h2'} 2024-09-15T20:15:01.143 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-15T20:15:01.167 INFO:teuthology.task.internal:roles: ubuntu@smithi078.front.sepia.ceph.com - ['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'] 2024-09-15T20:15:01.167 INFO:teuthology.task.internal:roles: ubuntu@smithi146.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0'] 2024-09-15T20:15:01.167 INFO:teuthology.run_tasks:Running task console_log... 2024-09-15T20:15:01.265 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fc89ad5f370>, signals=[15]) 2024-09-15T20:15:01.265 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-15T20:15:01.276 INFO:teuthology.task.internal:Opening connections... 2024-09-15T20:15:01.276 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:15:01.278 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:15:01.363 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:15:01.364 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:15:01.443 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-15T20:15:01.449 DEBUG:teuthology.orchestra.run.smithi078:> uname -m 2024-09-15T20:15:01.454 INFO:teuthology.orchestra.run.smithi078.stdout:x86_64 2024-09-15T20:15:01.454 DEBUG:teuthology.orchestra.run.smithi078:> cat /etc/os-release 2024-09-15T20:15:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-15T20:15:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:NAME="Ubuntu" 2024-09-15T20:15:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_ID="22.04" 2024-09-15T20:15:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-15T20:15:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:VERSION_CODENAME=jammy 2024-09-15T20:15:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:ID=ubuntu 2024-09-15T20:15:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:ID_LIKE=debian 2024-09-15T20:15:01.500 INFO:teuthology.orchestra.run.smithi078.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T20:15:01.501 INFO:teuthology.orchestra.run.smithi078.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T20:15:01.501 INFO:teuthology.orchestra.run.smithi078.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T20:15:01.501 INFO:teuthology.orchestra.run.smithi078.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T20:15:01.501 INFO:teuthology.orchestra.run.smithi078.stdout:UBUNTU_CODENAME=jammy 2024-09-15T20:15:01.501 INFO:teuthology.lock.ops:Updating smithi078.front.sepia.ceph.com on lock server 2024-09-15T20:15:01.521 DEBUG:teuthology.orchestra.run.smithi146:> uname -m 2024-09-15T20:15:01.526 INFO:teuthology.orchestra.run.smithi146.stdout:x86_64 2024-09-15T20:15:01.527 DEBUG:teuthology.orchestra.run.smithi146:> cat /etc/os-release 2024-09-15T20:15:01.572 INFO:teuthology.orchestra.run.smithi146.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-15T20:15:01.572 INFO:teuthology.orchestra.run.smithi146.stdout:NAME="Ubuntu" 2024-09-15T20:15:01.572 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_ID="22.04" 2024-09-15T20:15:01.572 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-15T20:15:01.573 INFO:teuthology.orchestra.run.smithi146.stdout:VERSION_CODENAME=jammy 2024-09-15T20:15:01.573 INFO:teuthology.orchestra.run.smithi146.stdout:ID=ubuntu 2024-09-15T20:15:01.573 INFO:teuthology.orchestra.run.smithi146.stdout:ID_LIKE=debian 2024-09-15T20:15:01.573 INFO:teuthology.orchestra.run.smithi146.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-15T20:15:01.573 INFO:teuthology.orchestra.run.smithi146.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-15T20:15:01.573 INFO:teuthology.orchestra.run.smithi146.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-15T20:15:01.573 INFO:teuthology.orchestra.run.smithi146.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-15T20:15:01.573 INFO:teuthology.orchestra.run.smithi146.stdout:UBUNTU_CODENAME=jammy 2024-09-15T20:15:01.573 INFO:teuthology.lock.ops:Updating smithi146.front.sepia.ceph.com on lock server 2024-09-15T20:15:01.596 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-15T20:15:01.605 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-15T20:15:01.617 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-15T20:15:01.617 DEBUG:teuthology.orchestra.run.smithi078:> test '!' -e /home/ubuntu/cephtest 2024-09-15T20:15:01.620 DEBUG:teuthology.orchestra.run.smithi146:> test '!' -e /home/ubuntu/cephtest 2024-09-15T20:15:01.624 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-15T20:15:01.629 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-15T20:15:01.629 DEBUG:teuthology.orchestra.run.smithi078:> test -z $(ls -A /var/lib/ceph) 2024-09-15T20:15:01.666 DEBUG:teuthology.orchestra.run.smithi146:> test -z $(ls -A /var/lib/ceph) 2024-09-15T20:15:01.745 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-15T20:15:01.792 INFO:teuthology.run_tasks:Running task kernel... 2024-09-15T20:15:01.806 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-15T20:15:01.806 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-15T20:15:01.806 DEBUG:teuthology.orchestra.run.smithi078:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T20:15:01.807 DEBUG:teuthology.orchestra.run.smithi146:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-15T20:15:01.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:15:01.811 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-09-15T20:15:01.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:15:01.812 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-09-15T20:15:01.856 INFO:teuthology.orchestra.run.smithi078.stdout:5.15.0-56-generic 2024-09-15T20:15:01.856 INFO:teuthology.task.kernel:Running kernel on smithi078: 5.15.0-56-generic 2024-09-15T20:15:01.856 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2024-09-15T20:15:01.860 INFO:teuthology.orchestra.run.smithi146.stdout:5.15.0-56-generic 2024-09-15T20:15:01.860 INFO:teuthology.task.kernel:Running kernel on smithi146: 5.15.0-56-generic 2024-09-15T20:15:01.860 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2024-09-15T20:15:01.980 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-09-15T20:15:02.007 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-09-15T20:15:02.114 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:15:02.131 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-15T20:15:02.149 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:15:02.159 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-15T20:15:02.196 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-15T20:15:02.224 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-15T20:15:02.230 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-15T20:15:02.262 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-15T20:15:02.331 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-15T20:15:02.364 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-15T20:15:02.404 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [2,023 kB] 2024-09-15T20:15:02.442 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-15T20:15:02.449 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-15T20:15:02.449 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-15T20:15:02.450 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-15T20:15:02.466 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [696 kB] 2024-09-15T20:15:02.471 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [352 kB] 2024-09-15T20:15:02.473 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-15T20:15:02.474 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-15T20:15:02.474 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,437 kB] 2024-09-15T20:15:02.480 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-15T20:15:02.485 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-15T20:15:02.485 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-15T20:15:02.494 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [419 kB] 2024-09-15T20:15:02.494 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-15T20:15:02.498 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-15T20:15:02.498 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [732 kB] 2024-09-15T20:15:02.500 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-15T20:15:02.502 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-15T20:15:02.503 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-15T20:15:02.505 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,124 kB] 2024-09-15T20:15:02.515 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-15T20:15:02.517 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.1 kB] 2024-09-15T20:15:02.517 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-15T20:15:02.518 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-15T20:15:02.520 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-15T20:15:02.521 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-15T20:15:02.521 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-15T20:15:02.521 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-15T20:15:02.521 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-15T20:15:02.522 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-15T20:15:02.534 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-15T20:15:02.534 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-15T20:15:02.553 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-15T20:15:02.553 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-15T20:15:02.553 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-15T20:15:02.553 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-15T20:15:02.553 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-15T20:15:02.554 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-15T20:15:02.554 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-15T20:15:02.555 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-15T20:15:02.555 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-15T20:15:02.555 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-15T20:15:02.556 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-15T20:15:02.570 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-15T20:15:02.587 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-15T20:15:02.673 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-15T20:15:02.702 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-15T20:15:02.987 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,806 kB] 2024-09-15T20:15:03.096 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [539 kB] 2024-09-15T20:15:03.112 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-15T20:15:03.121 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-15T20:15:03.122 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-15T20:15:03.123 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-15T20:15:03.160 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [295 kB] 2024-09-15T20:15:03.172 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-15T20:15:03.173 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,377 kB] 2024-09-15T20:15:03.219 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-15T20:15:03.230 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-15T20:15:03.231 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-15T20:15:03.254 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-15T20:15:03.258 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-15T20:15:03.259 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [409 kB] 2024-09-15T20:15:03.267 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-15T20:15:03.268 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [627 kB] 2024-09-15T20:15:03.273 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-15T20:15:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-15T20:15:03.279 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-15T20:15:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-15T20:15:03.280 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-15T20:15:03.287 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [902 kB] 2024-09-15T20:15:03.318 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-15T20:15:03.324 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-15T20:15:03.324 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-15T20:15:03.324 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-15T20:15:03.358 INFO:teuthology.orchestra.run.smithi078.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-15T20:15:03.406 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-15T20:15:03.406 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-15T20:15:08.779 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 16.0 MB in 2s (7,595 kB/s) 2024-09-15T20:15:08.945 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 16.0 MB in 2s (7,486 kB/s) 2024-09-15T20:15:10.161 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-09-15T20:15:10.179 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T20:15:10.231 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-09-15T20:15:10.360 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-15T20:15:10.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T20:15:10.416 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-09-15T20:15:10.417 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-09-15T20:15:10.431 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-15T20:15:10.583 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:15:10.583 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:15:10.583 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:15:10.584 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:15:10.584 INFO:teuthology.orchestra.run.smithi078.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-15T20:15:10.584 INFO:teuthology.orchestra.run.smithi078.stdout: python2.7-minimal 2024-09-15T20:15:10.584 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:15:10.585 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2024-09-15T20:15:10.585 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-15T20:15:10.585 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-15T20:15:10.586 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-15T20:15:10.587 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2024-09-15T20:15:10.587 INFO:teuthology.orchestra.run.smithi078.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-15T20:15:10.587 INFO:teuthology.orchestra.run.smithi078.stdout:Recommended packages: 2024-09-15T20:15:10.587 INFO:teuthology.orchestra.run.smithi078.stdout: thermald 2024-09-15T20:15:10.616 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-09-15T20:15:10.617 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-09-15T20:15:10.635 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2024-09-15T20:15:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-15T20:15:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-15T20:15:10.636 INFO:teuthology.orchestra.run.smithi078.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-15T20:15:10.637 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages will be upgraded: 2024-09-15T20:15:10.638 INFO:teuthology.orchestra.run.smithi078.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T20:15:10.691 INFO:teuthology.orchestra.run.smithi078.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-15T20:15:10.691 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 113 MB of archives. 2024-09-15T20:15:10.691 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-15T20:15:10.691 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-15T20:15:10.787 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:15:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:15:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:15:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:15:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-15T20:15:10.788 INFO:teuthology.orchestra.run.smithi146.stdout: python2.7-minimal 2024-09-15T20:15:10.788 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:15:10.789 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2024-09-15T20:15:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: linux-generic linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-15T20:15:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: linux-headers-generic linux-image-5.15.0-121-generic 2024-09-15T20:15:10.789 INFO:teuthology.orchestra.run.smithi146.stdout: linux-modules-5.15.0-121-generic linux-modules-extra-5.15.0-121-generic 2024-09-15T20:15:10.791 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2024-09-15T20:15:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-15T20:15:10.791 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2024-09-15T20:15:10.791 INFO:teuthology.orchestra.run.smithi146.stdout: thermald 2024-09-15T20:15:10.843 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2024-09-15T20:15:10.844 INFO:teuthology.orchestra.run.smithi146.stdout: linux-headers-5.15.0-121 linux-headers-5.15.0-121-generic 2024-09-15T20:15:10.844 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.15.0-121-generic linux-modules-5.15.0-121-generic 2024-09-15T20:15:10.844 INFO:teuthology.orchestra.run.smithi146.stdout: linux-modules-extra-5.15.0-121-generic 2024-09-15T20:15:10.845 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages will be upgraded: 2024-09-15T20:15:10.845 INFO:teuthology.orchestra.run.smithi146.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-15T20:15:11.025 INFO:teuthology.orchestra.run.smithi146.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-15T20:15:11.026 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 113 MB of archives. 2024-09-15T20:15:11.026 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 584 MB of additional disk space will be used. 2024-09-15T20:15:11.026 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-121-generic amd64 5.15.0-121.131 [22.7 MB] 2024-09-15T20:15:11.103 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-15T20:15:11.193 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-15T20:15:11.962 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-15T20:15:11.962 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-15T20:15:11.962 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-15T20:15:12.079 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-15T20:15:12.171 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-15T20:15:12.548 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-121-generic amd64 5.15.0-121.131 [11.5 MB] 2024-09-15T20:15:12.573 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 113 MB in 2s (74.2 MB/s) 2024-09-15T20:15:12.758 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-15T20:15:12.898 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-121-generic amd64 5.15.0-121.131 [63.9 MB] 2024-09-15T20:15:14.842 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-15T20:15:14.846 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:15:15.004 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:15.174 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.121.121 [1,694 B] 2024-09-15T20:15:15.175 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.121.121 [2,464 B] 2024-09-15T20:15:15.175 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121 all 5.15.0-121.131 [12.3 MB] 2024-09-15T20:15:15.710 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-121-generic amd64 5.15.0-121.131 [2,827 kB] 2024-09-15T20:15:15.828 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.121.121 [2,316 B] 2024-09-15T20:15:16.252 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 113 MB in 5s (22.8 MB/s) 2024-09-15T20:15:16.417 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-modules-5.15.0-121-generic. 2024-09-15T20:15:17.113 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-15T20:15:17.310 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:15:17.310 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:17.668 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-15T20:15:17.687 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:15:17.724 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:18.333 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-15T20:15:18.337 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../0-linux-modules-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:15:18.398 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:20.842 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-image-5.15.0-121-generic. 2024-09-15T20:15:20.863 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../1-linux-image-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:15:20.966 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:21.437 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-121-generic. 2024-09-15T20:15:21.457 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:15:21.495 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:23.808 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:15:23.935 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:15:24.277 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:15:24.397 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:15:24.659 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-15T20:15:24.669 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-15T20:15:24.700 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-15T20:15:27.609 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../3-linux-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:15:27.736 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:15:28.094 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:15:28.222 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-image-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:15:28.525 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-headers-5.15.0-121. 2024-09-15T20:15:28.536 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../5-linux-headers-5.15.0-121_5.15.0-121.131_all.deb ... 2024-09-15T20:15:28.575 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-15T20:15:29.766 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-15T20:15:29.804 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:15:29.841 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:32.987 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:15:33.096 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:15:33.421 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-15T20:15:33.542 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:33.668 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-15T20:15:33.796 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:34.369 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package linux-headers-5.15.0-121-generic. 2024-09-15T20:15:34.406 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../6-linux-headers-5.15.0-121-generic_5.15.0-121.131_amd64.deb ... 2024-09-15T20:15:34.443 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:35.082 INFO:teuthology.orchestra.run.smithi078.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:15:35.082 INFO:teuthology.orchestra.run.smithi078.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-15T20:15:35.083 INFO:teuthology.orchestra.run.smithi078.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:15:35.083 INFO:teuthology.orchestra.run.smithi078.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-15T20:15:35.225 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:36.608 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:37.097 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.121.121_amd64.deb ... 2024-09-15T20:15:37.220 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking linux-headers-generic (5.15.0.121.121) over (5.15.0.56.54) ... 2024-09-15T20:15:37.561 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-5.15.0-121 (5.15.0-121.131) ... 2024-09-15T20:15:38.377 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-15T20:15:38.503 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-15T20:15:38.517 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:38.621 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-headers-generic (5.15.0.121.121) ... 2024-09-15T20:15:38.629 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:38.698 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T20:15:38.698 INFO:teuthology.orchestra.run.smithi078.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:15:38.746 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:40.072 INFO:teuthology.orchestra.run.smithi146.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:15:40.072 INFO:teuthology.orchestra.run.smithi146.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-15T20:15:40.072 INFO:teuthology.orchestra.run.smithi146.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:15:40.072 INFO:teuthology.orchestra.run.smithi146.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-121-generic 2024-09-15T20:15:40.232 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-modules-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:41.632 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-modules-extra-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:43.017 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-image-generic (5.15.0.121.121) ... 2024-09-15T20:15:43.151 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up linux-generic (5.15.0.121.121) ... 2024-09-15T20:15:43.253 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for linux-image-5.15.0-121-generic (5.15.0-121.131) ... 2024-09-15T20:15:43.312 INFO:teuthology.orchestra.run.smithi146.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-15T20:15:43.312 INFO:teuthology.orchestra.run.smithi146.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T20:15:53.127 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub' 2024-09-15T20:15:53.137 INFO:teuthology.orchestra.run.smithi078.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T20:15:53.155 INFO:teuthology.orchestra.run.smithi078.stdout:Generating grub configuration file ... 2024-09-15T20:15:53.438 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:15:53.460 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:15:53.808 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:15:53.815 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-15T20:15:53.890 INFO:teuthology.orchestra.run.smithi078.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-15T20:15:53.895 INFO:teuthology.orchestra.run.smithi078.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-15T20:15:54.063 INFO:teuthology.orchestra.run.smithi078.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-15T20:15:54.064 INFO:teuthology.orchestra.run.smithi078.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-15T20:15:54.064 INFO:teuthology.orchestra.run.smithi078.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-15T20:15:54.099 INFO:teuthology.orchestra.run.smithi078.stdout:done 2024-09-15T20:15:54.584 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 21 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2024-09-15T20:15:54.603 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.15.0.121.121 2024-09-15T20:15:54.604 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-15T20:15:54.604 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T20:15:54.604 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2024-09-15T20:15:54.604 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2024-09-15T20:15:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2024-09-15T20:15:54.604 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2024-09-15T20:15:54.604 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-15T20:15:54.604 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T20:15:54.604 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T20:15:54.604 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-15T20:15:54.605 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi078.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T20:15:54.605 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get clean 2024-09-15T20:15:54.748 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-09-15T20:15:54.864 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:15:54.874 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-15T20:15:54.892 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-15T20:15:54.986 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:4 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-15T20:15:56.289 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-09-15T20:15:56.307 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T20:15:56.361 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-09-15T20:15:56.546 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-09-15T20:15:56.556 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-09-15T20:15:56.733 INFO:teuthology.orchestra.run.smithi078.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-15T20:15:56.734 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:15:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:15:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:15:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:15:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-15T20:15:56.734 INFO:teuthology.orchestra.run.smithi078.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-15T20:15:56.735 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:15:56.773 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-15T20:15:56.775 DEBUG:teuthology.orchestra.run.smithi078:> dpkg -s linux-image-generic 2024-09-15T20:15:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:Package: linux-image-generic 2024-09-15T20:15:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:Status: install ok installed 2024-09-15T20:15:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:Priority: optional 2024-09-15T20:15:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:Section: kernel 2024-09-15T20:15:56.834 INFO:teuthology.orchestra.run.smithi078.stdout:Installed-Size: 21 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout:Architecture: amd64 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout:Source: linux-meta 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout:Version: 5.15.0.121.121 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout:Recommends: thermald 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout:Description: Generic Linux kernel image 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout: This package will always depend on the latest generic kernel image 2024-09-15T20:15:56.835 INFO:teuthology.orchestra.run.smithi078.stdout: available. 2024-09-15T20:15:56.836 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-09-15T20:15:56.879 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.Cp4DzwxYIt 2024-09-15T20:15:56.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo cp /boot/grub/grub.cfg /tmp/tmp.Cp4DzwxYIt 2024-09-15T20:15:56.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.Cp4DzwxYIt 2024-09-15T20:15:57.056 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.Cp4DzwxYIt is 10KB 2024-09-15T20:15:57.069 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.Cp4DzwxYIt 2024-09-15T20:15:57.074 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T20:15:57.127 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:15:57.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T20:15:57.127 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T20:15:57.205 INFO:teuthology.orchestra.run.smithi146.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-15T20:15:57.205 INFO:teuthology.orchestra.run.smithi146.stdout:Sourcing file `/etc/default/grub' 2024-09-15T20:15:57.206 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-15T20:15:57.206 DEBUG:teuthology.orchestra.run.smithi078:> sudo update-grub 2024-09-15T20:15:57.219 INFO:teuthology.orchestra.run.smithi146.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T20:15:57.232 INFO:teuthology.orchestra.run.smithi146.stdout:Generating grub configuration file ... 2024-09-15T20:15:57.508 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:15:57.523 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:15:57.810 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:15:57.816 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-15T20:15:57.887 INFO:teuthology.orchestra.run.smithi146.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-15T20:15:57.892 INFO:teuthology.orchestra.run.smithi146.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-15T20:15:58.060 INFO:teuthology.orchestra.run.smithi146.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-15T20:15:58.061 INFO:teuthology.orchestra.run.smithi146.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-15T20:15:58.061 INFO:teuthology.orchestra.run.smithi146.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-15T20:15:58.100 INFO:teuthology.orchestra.run.smithi146.stdout:done 2024-09-15T20:15:58.386 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub' 2024-09-15T20:15:58.388 INFO:teuthology.orchestra.run.smithi078.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T20:15:58.389 INFO:teuthology.orchestra.run.smithi078.stderr:Generating grub configuration file ... 2024-09-15T20:15:58.566 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:15:58.572 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:15:58.575 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2024-09-15T20:15:58.591 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2024-09-15T20:15:58.591 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2024-09-15T20:15:58.591 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2024-09-15T20:15:58.591 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2024-09-15T20:15:58.591 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 21 2024-09-15T20:15:58.591 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T20:15:58.591 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2024-09-15T20:15:58.591 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2024-09-15T20:15:58.592 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.15.0.121.121 2024-09-15T20:15:58.592 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-15T20:15:58.592 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T20:15:58.592 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2024-09-15T20:15:58.592 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2024-09-15T20:15:58.592 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2024-09-15T20:15:58.592 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2024-09-15T20:15:58.592 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-121-generic 2024-09-15T20:15:58.592 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-15T20:15:58.592 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-15T20:15:58.593 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-15T20:15:58.593 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi146.front.sepia.ceph.com, path=None, version=distro) 2024-09-15T20:15:58.593 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get clean 2024-09-15T20:15:58.768 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-09-15T20:15:58.809 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:15:58.815 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-15T20:15:58.855 INFO:teuthology.orchestra.run.smithi078.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-15T20:15:58.859 INFO:teuthology.orchestra.run.smithi078.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-15T20:15:58.904 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-15T20:15:58.904 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:15:58.913 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-15T20:15:58.930 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-15T20:15:58.999 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-15T20:15:58.999 INFO:teuthology.orchestra.run.smithi078.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-15T20:15:58.999 INFO:teuthology.orchestra.run.smithi078.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-15T20:15:59.024 INFO:teuthology.orchestra.run.smithi078.stderr:done 2024-09-15T20:15:59.033 DEBUG:teuthology.orchestra.run.smithi078:> sudo shutdown -r now 2024-09-15T20:16:00.316 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-15T20:16:00.334 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-15T20:16:00.388 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-15T20:16:00.575 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-09-15T20:16:00.575 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-09-15T20:16:00.750 INFO:teuthology.orchestra.run.smithi146.stdout:linux-image-generic is already the newest version (5.15.0.121.121). 2024-09-15T20:16:00.751 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:16:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:16:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:16:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:16:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-15T20:16:00.751 INFO:teuthology.orchestra.run.smithi146.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-15T20:16:00.751 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:16:00.792 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-15T20:16:00.794 DEBUG:teuthology.orchestra.run.smithi146:> dpkg -s linux-image-generic 2024-09-15T20:16:00.847 INFO:teuthology.orchestra.run.smithi146.stdout:Package: linux-image-generic 2024-09-15T20:16:00.847 INFO:teuthology.orchestra.run.smithi146.stdout:Status: install ok installed 2024-09-15T20:16:00.847 INFO:teuthology.orchestra.run.smithi146.stdout:Priority: optional 2024-09-15T20:16:00.847 INFO:teuthology.orchestra.run.smithi146.stdout:Section: kernel 2024-09-15T20:16:00.847 INFO:teuthology.orchestra.run.smithi146.stdout:Installed-Size: 21 2024-09-15T20:16:00.847 INFO:teuthology.orchestra.run.smithi146.stdout:Maintainer: Ubuntu Kernel Team 2024-09-15T20:16:00.847 INFO:teuthology.orchestra.run.smithi146.stdout:Architecture: amd64 2024-09-15T20:16:00.848 INFO:teuthology.orchestra.run.smithi146.stdout:Source: linux-meta 2024-09-15T20:16:00.848 INFO:teuthology.orchestra.run.smithi146.stdout:Version: 5.15.0.121.121 2024-09-15T20:16:00.848 INFO:teuthology.orchestra.run.smithi146.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-121), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-15T20:16:00.848 INFO:teuthology.orchestra.run.smithi146.stdout:Depends: linux-image-5.15.0-121-generic, linux-modules-extra-5.15.0-121-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-15T20:16:00.848 INFO:teuthology.orchestra.run.smithi146.stdout:Recommends: thermald 2024-09-15T20:16:00.848 INFO:teuthology.orchestra.run.smithi146.stdout:Description: Generic Linux kernel image 2024-09-15T20:16:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: This package will always depend on the latest generic kernel image 2024-09-15T20:16:00.848 INFO:teuthology.orchestra.run.smithi146.stdout: available. 2024-09-15T20:16:00.848 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-09-15T20:16:00.895 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.NyapxYDmIU 2024-09-15T20:16:00.896 DEBUG:teuthology.orchestra.run.smithi146:> sudo cp /boot/grub/grub.cfg /tmp/tmp.NyapxYDmIU 2024-09-15T20:16:00.952 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.NyapxYDmIU 2024-09-15T20:16:01.075 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.NyapxYDmIU is 10KB 2024-09-15T20:16:01.089 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.NyapxYDmIU 2024-09-15T20:16:01.094 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-15T20:16:01.147 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:16:01.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-15T20:16:01.147 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-15T20:16:01.215 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-121-generic 2024-09-15T20:16:01.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo update-grub 2024-09-15T20:16:02.429 INFO:teuthology.orchestra.run.smithi146.stderr:Sourcing file `/etc/default/grub' 2024-09-15T20:16:02.430 INFO:teuthology.orchestra.run.smithi146.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-15T20:16:02.433 INFO:teuthology.orchestra.run.smithi146.stderr:Generating grub configuration file ... 2024-09-15T20:16:02.613 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.15.0-121-generic 2024-09-15T20:16:02.621 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.15.0-121-generic 2024-09-15T20:16:02.861 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-15T20:16:02.865 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-15T20:16:02.901 INFO:teuthology.orchestra.run.smithi146.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-15T20:16:02.906 INFO:teuthology.orchestra.run.smithi146.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-15T20:16:03.026 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-15T20:16:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-15T20:16:03.027 INFO:teuthology.orchestra.run.smithi146.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-15T20:16:03.054 INFO:teuthology.orchestra.run.smithi146.stderr:done 2024-09-15T20:16:03.062 DEBUG:teuthology.orchestra.run.smithi146:> sudo shutdown -r now 2024-09-15T20:16:29.061 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-15T20:16:29.061 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T20:16:29.062 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:16:33.069 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-15T20:16:33.069 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-15T20:16:33.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:16:47.421 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-09-15T20:16:51.521 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-15T20:16:56.425 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T20:16:56.425 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:16:59.489 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.78 2024-09-15T20:17:00.524 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-15T20:17:00.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:17:03.581 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-15T20:17:11.497 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T20:17:11.498 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:17:15.585 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-15T20:17:15.586 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:17:43.406 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-09-15T20:17:44.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T20:17:44.118 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-121-generic"... 2024-09-15T20:17:44.119 DEBUG:teuthology.orchestra.run.smithi078:> uname -r 2024-09-15T20:17:44.163 INFO:teuthology.orchestra.run.smithi078.stdout:5.15.0-121-generic 2024-09-15T20:17:44.163 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-15T20:17:44.163 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T20:17:44.163 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T20:17:45.164 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-15T20:17:45.165 DEBUG:teuthology.orchestra.run.smithi078:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T20:17:45.245 INFO:teuthology.orchestra.run.smithi078.stdout:ttyS1 2024-09-15T20:17:45.274 DEBUG:teuthology.parallel:result is None 2024-09-15T20:17:47.069 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.146 2024-09-15T20:18:02.070 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-15T20:18:02.071 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:18:02.318 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-09-15T20:18:02.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-15T20:18:02.886 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.15.0-121-generic"... 2024-09-15T20:18:02.887 DEBUG:teuthology.orchestra.run.smithi146:> uname -r 2024-09-15T20:18:02.933 INFO:teuthology.orchestra.run.smithi146.stdout:5.15.0-121-generic 2024-09-15T20:18:02.934 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-121-generic vs 5.15.0-121-generic 2024-09-15T20:18:02.934 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-15T20:18:02.934 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-15T20:18:03.935 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-15T20:18:03.936 DEBUG:teuthology.orchestra.run.smithi146:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-15T20:18:04.000 INFO:teuthology.orchestra.run.smithi146.stdout:ttyS1 2024-09-15T20:18:04.035 DEBUG:teuthology.parallel:result is None 2024-09-15T20:18:04.035 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-15T20:18:04.042 INFO:teuthology.task.internal:Creating test directory... 2024-09-15T20:18:04.042 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T20:18:04.045 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-15T20:18:04.050 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-15T20:18:04.063 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-15T20:18:04.072 INFO:teuthology.task.internal:Creating archive directory... 2024-09-15T20:18:04.073 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T20:18:04.096 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-15T20:18:04.124 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-15T20:18:04.133 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-15T20:18:04.133 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T20:18:04.156 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-15T20:18:04.176 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:18:04.177 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:18:04.184 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:18:04.186 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-15T20:18:04.187 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-15T20:18:04.194 INFO:teuthology.task.internal:Configuring sudo... 2024-09-15T20:18:04.195 DEBUG:teuthology.orchestra.run.smithi078:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T20:18:04.228 DEBUG:teuthology.orchestra.run.smithi146:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-15T20:18:04.242 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-15T20:18:04.254 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-15T20:18:04.254 DEBUG:teuthology.orchestra.run.smithi078:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T20:18:04.287 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-15T20:18:04.292 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:18:04.336 DEBUG:teuthology.orchestra.run.smithi078:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:18:04.383 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:18:04.384 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T20:18:04.441 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-15T20:18:04.448 DEBUG:teuthology.orchestra.run.smithi146:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-15T20:18:04.494 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:18:04.495 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-15T20:18:04.580 DEBUG:teuthology.orchestra.run.smithi078:> sudo service rsyslog restart 2024-09-15T20:18:04.582 DEBUG:teuthology.orchestra.run.smithi146:> sudo service rsyslog restart 2024-09-15T20:18:04.649 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-15T20:18:04.656 INFO:teuthology.task.internal:Starting timer... 2024-09-15T20:18:04.657 INFO:teuthology.run_tasks:Running task pcp... 2024-09-15T20:18:04.734 INFO:teuthology.run_tasks:Running task selinux... 2024-09-15T20:18:04.771 DEBUG:teuthology.task.selinux:Excluding smithi078: OS 'ubuntu' does not support SELinux 2024-09-15T20:18:04.771 DEBUG:teuthology.task.selinux:Excluding smithi146: OS 'ubuntu' does not support SELinux 2024-09-15T20:18:04.771 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-15T20:18:04.771 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-15T20:18:04.771 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-15T20:18:04.771 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-15T20:18:04.788 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-15T20:18:04.821 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-15T20:18:04.822 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi078.front.sepia.ceph.com,smithi146.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-15T20:26:40.007 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi078.front.sepia.ceph.com'), Remote(name='ubuntu@smithi146.front.sepia.ceph.com')] 2024-09-15T20:26:40.008 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T20:26:40.009 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi078.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:26:40.092 DEBUG:teuthology.orchestra.run.smithi078:> true 2024-09-15T20:26:40.165 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi078.front.sepia.ceph.com' 2024-09-15T20:26:40.165 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-15T20:26:40.165 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi146.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-15T20:26:40.247 DEBUG:teuthology.orchestra.run.smithi146:> true 2024-09-15T20:26:40.325 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi146.front.sepia.ceph.com' 2024-09-15T20:26:40.326 INFO:teuthology.run_tasks:Running task clock... 2024-09-15T20:26:40.334 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-15T20:26:40.335 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T20:26:40.335 DEBUG:teuthology.orchestra.run.smithi078:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:26:40.337 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-15T20:26:40.338 DEBUG:teuthology.orchestra.run.smithi146:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:26:40.365 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-15T20:26:40.365 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Command line: ntpd -gq 2024-09-15T20:26:40.365 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: ---------------------------------------------------- 2024-09-15T20:26:40.366 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: ntp-4 is maintained by Network Time Foundation, 2024-09-15T20:26:40.366 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-15T20:26:40.366 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: corporation. Support and training for ntp-4 are 2024-09-15T20:26:40.366 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: available at https://www.nwtime.org/support 2024-09-15T20:26:40.366 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: ---------------------------------------------------- 2024-09-15T20:26:40.366 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: proto: precision = 0.045 usec (-24) 2024-09-15T20:26:40.366 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: basedate set to 2022-02-04 2024-09-15T20:26:40.367 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: gps base set to 2022-02-06 (week 2196) 2024-09-15T20:26:40.367 INFO:teuthology.orchestra.run.smithi078.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T20:26:40.367 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T20:26:40.367 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T20:26:40.368 INFO:teuthology.orchestra.run.smithi078.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T20:26:40.368 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T20:26:40.368 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T20:26:40.369 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T20:26:40.369 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Listen normally on 3 enp3s0f1 172.21.15.78:123 2024-09-15T20:26:40.369 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Listen normally on 4 lo [::1]:123 2024-09-15T20:26:40.369 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d4dd%5]:123 2024-09-15T20:26:40.369 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:40 ntpd[18467]: Listening on routing socket on fd #22 for interface updates 2024-09-15T20:26:40.395 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-15T20:26:40.395 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Command line: ntpd -gq 2024-09-15T20:26:40.395 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: ---------------------------------------------------- 2024-09-15T20:26:40.395 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: ntp-4 is maintained by Network Time Foundation, 2024-09-15T20:26:40.395 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-15T20:26:40.395 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: corporation. Support and training for ntp-4 are 2024-09-15T20:26:40.395 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: available at https://www.nwtime.org/support 2024-09-15T20:26:40.395 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: ---------------------------------------------------- 2024-09-15T20:26:40.396 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: proto: precision = 0.043 usec (-24) 2024-09-15T20:26:40.396 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: basedate set to 2022-02-04 2024-09-15T20:26:40.396 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: gps base set to 2022-02-06 (week 2196) 2024-09-15T20:26:40.396 INFO:teuthology.orchestra.run.smithi146.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T20:26:40.397 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-15T20:26:40.397 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: restrict ::: KOD does nothing without LIMITED. 2024-09-15T20:26:40.397 INFO:teuthology.orchestra.run.smithi146.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-15T20:26:40.398 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Listen and drop on 0 v6wildcard [::]:123 2024-09-15T20:26:40.398 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-15T20:26:40.398 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Listen normally on 2 lo 127.0.0.1:123 2024-09-15T20:26:40.398 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Listen normally on 3 enp3s0f1 172.21.15.146:123 2024-09-15T20:26:40.398 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Listen normally on 4 lo [::1]:123 2024-09-15T20:26:40.398 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:76fd%5]:123 2024-09-15T20:26:40.398 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:40 ntpd[18500]: Listening on routing socket on fd #22 for interface updates 2024-09-15T20:26:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:41 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:41 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:41 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:41 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:41 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:41 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:42.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:42 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:42.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:42 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:42.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:42 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:42.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:42 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:42.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:42 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:42.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:42 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:42.395 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:42 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:42.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:42 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:42.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:42 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:42.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:42 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:42.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:42 ntpd[18500]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:42.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:42 ntpd[18500]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:43 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:43 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:43 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:43 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:43 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:43 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:43 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:43.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:43 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:43.395 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:43 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:43.395 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:43 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:44.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:44.368 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:44 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:44.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:44.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:44.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:44.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:44.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:44.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:44.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:44.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:44.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:44 ntpd[18500]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:45.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:45 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:45.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:45 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:45.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:45 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:45.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:45 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:45.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:45 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:45.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:45 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:45 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:45 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:45 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:45 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:45 ntpd[18500]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:45.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:45 ntpd[18500]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:46.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:46.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:46 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:46.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:46.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:46.398 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:46 ntpd[18500]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:47 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:47 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:47 ntpd[18467]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:47.366 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:47 ntpd[18467]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:47 ntpd[18467]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:47 ntpd[18467]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:47.367 INFO:teuthology.orchestra.run.smithi078.stdout:15 Sep 20:26:47 ntpd[18467]: ntpd: time slew +0.000169 s 2024-09-15T20:26:47.367 INFO:teuthology.orchestra.run.smithi078.stdout:ntpd: time slew +0.000169s 2024-09-15T20:26:47.367 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:47 ntpd[18467]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T20:26:47.368 INFO:teuthology.orchestra.run.smithi078.stderr:15 Sep 20:26:47 ntpd[18467]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T20:26:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:47 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:47 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:47 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:47 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:47 ntpd[18500]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:47.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:47 ntpd[18500]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T20:26:47.421 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2024-09-15T20:26:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:26:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:26:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:26:47.421 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:26:48.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:48.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:48.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:48.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:48.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-15T20:26:48.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: can't open /var/log/ntpstats/rawstats.20240915: Permission denied 2024-09-15T20:26:48.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-15T20:26:48.396 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: can't open /var/log/ntpstats/peerstats.20240915: Permission denied 2024-09-15T20:26:48.397 INFO:teuthology.orchestra.run.smithi146.stdout:15 Sep 20:26:48 ntpd[18500]: ntpd: time slew -0.001111 s 2024-09-15T20:26:48.397 INFO:teuthology.orchestra.run.smithi146.stdout:ntpd: time slew -0.001111s 2024-09-15T20:26:48.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-15T20:26:48.397 INFO:teuthology.orchestra.run.smithi146.stderr:15 Sep 20:26:48 ntpd[18500]: can't open /var/log/ntpstats/loopstats.20240915: Permission denied 2024-09-15T20:26:48.445 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T20:26:48.445 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2024-09-15T20:26:48.445 INFO:teuthology.orchestra.run.smithi146.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:26:48.446 INFO:teuthology.orchestra.run.smithi146.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:26:48.446 INFO:teuthology.orchestra.run.smithi146.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:26:48.446 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-15T20:26:48.447 INFO:teuthology.run_tasks:Running task install... 2024-09-15T20:26:48.459 DEBUG:teuthology.task.install:project ceph 2024-09-15T20:26:48.459 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}} 2024-09-15T20:26:48.459 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'} 2024-09-15T20:26:48.459 INFO:teuthology.task.install:Using flavor: default 2024-09-15T20:26:48.465 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-15T20:26:48.465 INFO:teuthology.task.install:extra packages: [] 2024-09-15T20:26:48.465 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-key list | grep Ceph 2024-09-15T20:26:48.466 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-key list | grep Ceph 2024-09-15T20:26:48.529 INFO:teuthology.orchestra.run.smithi078.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-15T20:26:48.544 INFO:teuthology.orchestra.run.smithi146.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-15T20:26:48.559 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T20:26:48.559 INFO:teuthology.orchestra.run.smithi078.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T20:26:48.561 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T20:26:48.561 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:26:48.571 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-15T20:26:48.571 INFO:teuthology.orchestra.run.smithi146.stdout:uid [ unknown] Ceph.com (release key) 2024-09-15T20:26:48.571 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2024-09-15T20:26:48.571 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:26:48.711 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-15T20:26:48.711 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:26:48.715 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default/ 2024-09-15T20:26:48.715 INFO:teuthology.task.install.deb:Package version is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:26:49.030 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:26:49.030 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T20:26:49.042 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:26:49.042 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-15T20:26:49.043 DEBUG:teuthology.orchestra.run.smithi078:> sudo apt-get update 2024-09-15T20:26:49.058 DEBUG:teuthology.orchestra.run.smithi146:> sudo apt-get update 2024-09-15T20:26:49.211 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:1 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-15T20:26:49.331 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:2 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:26:49.353 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-15T20:26:49.354 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-15T20:26:49.414 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-15T20:26:49.427 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-15T20:26:49.473 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-15T20:26:49.493 INFO:teuthology.orchestra.run.smithi078.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-15T20:26:49.494 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:4 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy InRelease 2024-09-15T20:26:49.508 INFO:teuthology.orchestra.run.smithi146.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-15T20:26:49.549 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-15T20:26:49.571 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release [7,689 B] 2024-09-15T20:26:49.627 INFO:teuthology.orchestra.run.smithi078.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-15T20:26:49.648 INFO:teuthology.orchestra.run.smithi146.stdout:Ign:7 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-15T20:26:49.703 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-15T20:26:49.725 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,427 B] 2024-09-15T20:26:49.783 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-15T20:26:49.802 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.1 kB] 2024-09-15T20:26:49.879 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 29.2 kB in 1s (45.9 kB/s) 2024-09-15T20:26:49.956 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 29.2 kB in 1s (41.2 kB/s) 2024-09-15T20:26:50.904 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-09-15T20:26:50.922 DEBUG:teuthology.orchestra.run.smithi078:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:26:50.976 INFO:teuthology.orchestra.run.smithi078.stdout:Reading package lists... 2024-09-15T20:26:50.981 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-15T20:26:50.998 DEBUG:teuthology.orchestra.run.smithi146:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.3.0-4930-gfc3e0bff-1jammy cephadm=19.3.0-4930-gfc3e0bff-1jammy ceph-mds=19.3.0-4930-gfc3e0bff-1jammy ceph-mgr=19.3.0-4930-gfc3e0bff-1jammy ceph-common=19.3.0-4930-gfc3e0bff-1jammy ceph-fuse=19.3.0-4930-gfc3e0bff-1jammy ceph-test=19.3.0-4930-gfc3e0bff-1jammy ceph-volume=19.3.0-4930-gfc3e0bff-1jammy radosgw=19.3.0-4930-gfc3e0bff-1jammy python3-rados=19.3.0-4930-gfc3e0bff-1jammy python3-rgw=19.3.0-4930-gfc3e0bff-1jammy python3-cephfs=19.3.0-4930-gfc3e0bff-1jammy python3-rbd=19.3.0-4930-gfc3e0bff-1jammy libcephfs2=19.3.0-4930-gfc3e0bff-1jammy libcephfs-dev=19.3.0-4930-gfc3e0bff-1jammy librados2=19.3.0-4930-gfc3e0bff-1jammy librbd1=19.3.0-4930-gfc3e0bff-1jammy rbd-fuse=19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:26:51.061 INFO:teuthology.orchestra.run.smithi146.stdout:Reading package lists... 2024-09-15T20:26:51.163 INFO:teuthology.orchestra.run.smithi078.stdout:Building dependency tree... 2024-09-15T20:26:51.163 INFO:teuthology.orchestra.run.smithi078.stdout:Reading state information... 2024-09-15T20:26:51.269 INFO:teuthology.orchestra.run.smithi146.stdout:Building dependency tree... 2024-09-15T20:26:51.270 INFO:teuthology.orchestra.run.smithi146.stdout:Reading state information... 2024-09-15T20:26:51.347 INFO:teuthology.orchestra.run.smithi078.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:26:51.347 INFO:teuthology.orchestra.run.smithi078.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:26:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:26:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:26:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-15T20:26:51.348 INFO:teuthology.orchestra.run.smithi078.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-15T20:26:51.348 INFO:teuthology.orchestra.run.smithi078.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:26:51.349 INFO:teuthology.orchestra.run.smithi078.stdout:The following additional packages will be installed: 2024-09-15T20:26:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T20:26:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-15T20:26:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-15T20:26:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-15T20:26:51.349 INFO:teuthology.orchestra.run.smithi078.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-15T20:26:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-15T20:26:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-15T20:26:51.350 INFO:teuthology.orchestra.run.smithi078.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-15T20:26:51.351 INFO:teuthology.orchestra.run.smithi078.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: xmlstarlet zip 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout:Suggested packages: 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-15T20:26:51.352 INFO:teuthology.orchestra.run.smithi078.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T20:26:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-15T20:26:51.353 INFO:teuthology.orchestra.run.smithi078.stdout:Recommended packages: 2024-09-15T20:26:51.353 INFO:teuthology.orchestra.run.smithi078.stdout: btrfs-tools 2024-09-15T20:26:51.457 INFO:teuthology.orchestra.run.smithi078.stdout:The following NEW packages will be installed: 2024-09-15T20:26:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T20:26:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T20:26:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-15T20:26:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-15T20:26:51.457 INFO:teuthology.orchestra.run.smithi078.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-15T20:26:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-15T20:26:51.458 INFO:teuthology.orchestra.run.smithi078.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-15T20:26:51.459 INFO:teuthology.orchestra.run.smithi078.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-15T20:26:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-15T20:26:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-15T20:26:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-15T20:26:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-15T20:26:51.460 INFO:teuthology.orchestra.run.smithi078.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-15T20:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout:The following packages were automatically installed and are no longer required: 2024-09-15T20:26:51.462 INFO:teuthology.orchestra.run.smithi146.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-15T20:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-15T20:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-15T20:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-15T20:26:51.463 INFO:teuthology.orchestra.run.smithi146.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-15T20:26:51.464 INFO:teuthology.orchestra.run.smithi146.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-15T20:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout:The following additional packages will be installed: 2024-09-15T20:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-15T20:26:51.465 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev libnbd0 liboath0 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: libradosstriper1 librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: libthrift-0.16.0 lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: pkg-config python-asyncssh-doc python-babel-localedata 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: python-pastedeploy-tpl python3-asyncssh python3-babel python3-bcrypt 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-certifi 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-idna 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.functools python3-jaraco.text python3-jinja2 python3-joblib 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-kubernetes python3-logutils python3-mako python3-markupsafe 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-natsort python3-openssl python3-paste python3-pastedeploy 2024-09-15T20:26:51.466 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pastescript python3-pecan python3-pluggy python3-portend 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-prettytable python3-psutil python3-py python3-pyasn1 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pyasn1-modules python3-pygments python3-pyinotify python3-pytest 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-routes 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-rsa python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout: xmlstarlet zip 2024-09-15T20:26:51.467 INFO:teuthology.orchestra.run.smithi146.stdout:Suggested packages: 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout:Recommended packages: 2024-09-15T20:26:51.468 INFO:teuthology.orchestra.run.smithi146.stdout: btrfs-tools 2024-09-15T20:26:51.571 INFO:teuthology.orchestra.run.smithi146.stdout:The following NEW packages will be installed: 2024-09-15T20:26:51.571 INFO:teuthology.orchestra.run.smithi146.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-15T20:26:51.571 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-15T20:26:51.571 INFO:teuthology.orchestra.run.smithi146.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-15T20:26:51.571 INFO:teuthology.orchestra.run.smithi146.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-15T20:26:51.571 INFO:teuthology.orchestra.run.smithi146.stdout: liblua5.3-dev libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2024-09-15T20:26:51.571 INFO:teuthology.orchestra.run.smithi146.stdout: libqt5dbus5 libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-15T20:26:51.572 INFO:teuthology.orchestra.run.smithi146.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-15T20:26:51.572 INFO:teuthology.orchestra.run.smithi146.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-15T20:26:51.572 INFO:teuthology.orchestra.run.smithi146.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-15T20:26:51.573 INFO:teuthology.orchestra.run.smithi146.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-15T20:26:51.574 INFO:teuthology.orchestra.run.smithi146.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-15T20:26:51.574 INFO:teuthology.orchestra.run.smithi146.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-15T20:26:51.574 INFO:teuthology.orchestra.run.smithi146.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-15T20:26:51.574 INFO:teuthology.orchestra.run.smithi146.stdout: radosgw rbd-fuse socat unzip xmlstarlet zip 2024-09-15T20:26:51.647 INFO:teuthology.orchestra.run.smithi078.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-15T20:26:51.647 INFO:teuthology.orchestra.run.smithi078.stdout:Need to get 244 MB of archives. 2024-09-15T20:26:51.648 INFO:teuthology.orchestra.run.smithi078.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-15T20:26:51.648 INFO:teuthology.orchestra.run.smithi078.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-15T20:26:51.759 INFO:teuthology.orchestra.run.smithi146.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-15T20:26:51.759 INFO:teuthology.orchestra.run.smithi146.stdout:Need to get 244 MB of archives. 2024-09-15T20:26:51.759 INFO:teuthology.orchestra.run.smithi146.stdout:After this operation, 909 MB of additional disk space will be used. 2024-09-15T20:26:51.759 INFO:teuthology.orchestra.run.smithi146.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-15T20:26:51.816 INFO:teuthology.orchestra.run.smithi078.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-15T20:26:51.926 INFO:teuthology.orchestra.run.smithi146.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,633 kB] 2024-09-15T20:26:52.072 INFO:teuthology.orchestra.run.smithi078.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-15T20:26:52.084 INFO:teuthology.orchestra.run.smithi078.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-15T20:26:52.166 INFO:teuthology.orchestra.run.smithi078.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-15T20:26:52.179 INFO:teuthology.orchestra.run.smithi146.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-15T20:26:52.192 INFO:teuthology.orchestra.run.smithi146.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-15T20:26:52.273 INFO:teuthology.orchestra.run.smithi146.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-15T20:26:52.395 INFO:teuthology.orchestra.run.smithi078.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-15T20:26:52.406 INFO:teuthology.orchestra.run.smithi078.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-15T20:26:52.436 INFO:teuthology.orchestra.run.smithi078.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-15T20:26:52.445 INFO:teuthology.orchestra.run.smithi078.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-15T20:26:52.447 INFO:teuthology.orchestra.run.smithi078.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-15T20:26:52.448 INFO:teuthology.orchestra.run.smithi078.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-15T20:26:52.450 INFO:teuthology.orchestra.run.smithi078.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-15T20:26:52.470 INFO:teuthology.orchestra.run.smithi078.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-15T20:26:52.474 INFO:teuthology.orchestra.run.smithi078.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-15T20:26:52.478 INFO:teuthology.orchestra.run.smithi078.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-15T20:26:52.502 INFO:teuthology.orchestra.run.smithi146.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-15T20:26:52.514 INFO:teuthology.orchestra.run.smithi146.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-15T20:26:52.543 INFO:teuthology.orchestra.run.smithi146.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-15T20:26:52.552 INFO:teuthology.orchestra.run.smithi078.stdout:Get:16 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-15T20:26:52.554 INFO:teuthology.orchestra.run.smithi146.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2024-09-15T20:26:52.557 INFO:teuthology.orchestra.run.smithi146.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-15T20:26:52.558 INFO:teuthology.orchestra.run.smithi146.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-15T20:26:52.559 INFO:teuthology.orchestra.run.smithi078.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-15T20:26:52.559 INFO:teuthology.orchestra.run.smithi146.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-15T20:26:52.560 INFO:teuthology.orchestra.run.smithi078.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-15T20:26:52.564 INFO:teuthology.orchestra.run.smithi078.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-15T20:26:52.568 INFO:teuthology.orchestra.run.smithi078.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-15T20:26:52.571 INFO:teuthology.orchestra.run.smithi078.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-15T20:26:52.573 INFO:teuthology.orchestra.run.smithi078.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-15T20:26:52.574 INFO:teuthology.orchestra.run.smithi078.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-15T20:26:52.576 INFO:teuthology.orchestra.run.smithi146.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-15T20:26:52.580 INFO:teuthology.orchestra.run.smithi146.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-15T20:26:52.584 INFO:teuthology.orchestra.run.smithi146.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-15T20:26:52.641 INFO:teuthology.orchestra.run.smithi078.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-15T20:26:52.641 INFO:teuthology.orchestra.run.smithi078.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-15T20:26:52.642 INFO:teuthology.orchestra.run.smithi078.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-15T20:26:52.666 INFO:teuthology.orchestra.run.smithi146.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-15T20:26:52.722 INFO:teuthology.orchestra.run.smithi078.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-15T20:26:52.919 INFO:teuthology.orchestra.run.smithi078.stdout:Get:28 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-15T20:26:52.991 INFO:teuthology.orchestra.run.smithi078.stdout:Get:29 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-15T20:26:53.057 INFO:teuthology.orchestra.run.smithi078.stdout:Get:30 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-15T20:26:53.058 INFO:teuthology.orchestra.run.smithi078.stdout:Get:31 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-15T20:26:53.118 INFO:teuthology.orchestra.run.smithi078.stdout:Get:32 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-15T20:26:53.121 INFO:teuthology.orchestra.run.smithi078.stdout:Get:33 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-15T20:26:53.129 INFO:teuthology.orchestra.run.smithi078.stdout:Get:34 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-15T20:26:53.166 INFO:teuthology.orchestra.run.smithi146.stdout:Get:17 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.3.0-4930-gfc3e0bff-1jammy [3,334 kB] 2024-09-15T20:26:53.613 INFO:teuthology.orchestra.run.smithi146.stdout:Get:18 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.3.0-4930-gfc3e0bff-1jammy [721 kB] 2024-09-15T20:26:53.736 INFO:teuthology.orchestra.run.smithi146.stdout:Get:19 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.3.0-4930-gfc3e0bff-1jammy [356 kB] 2024-09-15T20:26:53.792 INFO:teuthology.orchestra.run.smithi146.stdout:Get:20 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.3.0-4930-gfc3e0bff-1jammy [32.8 kB] 2024-09-15T20:26:53.799 INFO:teuthology.orchestra.run.smithi146.stdout:Get:21 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.3.0-4930-gfc3e0bff-1jammy [180 kB] 2024-09-15T20:26:53.803 INFO:teuthology.orchestra.run.smithi146.stdout:Get:22 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.3.0-4930-gfc3e0bff-1jammy [71.3 kB] 2024-09-15T20:26:53.805 INFO:teuthology.orchestra.run.smithi146.stdout:Get:23 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.3.0-4930-gfc3e0bff-1jammy [338 kB] 2024-09-15T20:26:53.866 INFO:teuthology.orchestra.run.smithi146.stdout:Get:24 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.3.0-4930-gfc3e0bff-1jammy [7,966 kB] 2024-09-15T20:26:54.041 INFO:teuthology.orchestra.run.smithi078.stdout:Get:35 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-15T20:26:54.043 INFO:teuthology.orchestra.run.smithi078.stdout:Get:36 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-15T20:26:54.107 INFO:teuthology.orchestra.run.smithi078.stdout:Get:37 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-15T20:26:54.585 INFO:teuthology.orchestra.run.smithi146.stdout:Get:25 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.3.0-4930-gfc3e0bff-1jammy [112 kB] 2024-09-15T20:26:54.587 INFO:teuthology.orchestra.run.smithi146.stdout:Get:26 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.3.0-4930-gfc3e0bff-1jammy [473 kB] 2024-09-15T20:26:54.651 INFO:teuthology.orchestra.run.smithi146.stdout:Get:27 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.3.0-4930-gfc3e0bff-1jammy [28.2 MB] 2024-09-15T20:26:55.277 INFO:teuthology.orchestra.run.smithi146.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-15T20:26:55.279 INFO:teuthology.orchestra.run.smithi078.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-15T20:26:55.279 INFO:teuthology.orchestra.run.smithi078.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-15T20:26:55.280 INFO:teuthology.orchestra.run.smithi078.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-15T20:26:55.282 INFO:teuthology.orchestra.run.smithi146.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-15T20:26:55.286 INFO:teuthology.orchestra.run.smithi146.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-15T20:26:55.286 INFO:teuthology.orchestra.run.smithi078.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-15T20:26:55.288 INFO:teuthology.orchestra.run.smithi078.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-15T20:26:55.288 INFO:teuthology.orchestra.run.smithi078.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-15T20:26:55.289 INFO:teuthology.orchestra.run.smithi078.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-15T20:26:55.290 INFO:teuthology.orchestra.run.smithi146.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-15T20:26:55.291 INFO:teuthology.orchestra.run.smithi146.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-15T20:26:55.292 INFO:teuthology.orchestra.run.smithi146.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-15T20:26:55.292 INFO:teuthology.orchestra.run.smithi078.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T20:26:55.292 INFO:teuthology.orchestra.run.smithi078.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-15T20:26:55.292 INFO:teuthology.orchestra.run.smithi078.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-15T20:26:55.294 INFO:teuthology.orchestra.run.smithi146.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-15T20:26:55.294 INFO:teuthology.orchestra.run.smithi146.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-15T20:26:55.295 INFO:teuthology.orchestra.run.smithi146.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-15T20:26:55.296 INFO:teuthology.orchestra.run.smithi146.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-15T20:26:55.366 INFO:teuthology.orchestra.run.smithi078.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-15T20:26:55.367 INFO:teuthology.orchestra.run.smithi146.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-15T20:26:55.377 INFO:teuthology.orchestra.run.smithi078.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-15T20:26:55.378 INFO:teuthology.orchestra.run.smithi078.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-15T20:26:55.380 INFO:teuthology.orchestra.run.smithi146.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-15T20:26:55.380 INFO:teuthology.orchestra.run.smithi146.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-15T20:26:55.385 INFO:teuthology.orchestra.run.smithi146.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-15T20:26:55.386 INFO:teuthology.orchestra.run.smithi146.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-15T20:26:55.388 INFO:teuthology.orchestra.run.smithi146.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-15T20:26:55.388 INFO:teuthology.orchestra.run.smithi146.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-15T20:26:55.389 INFO:teuthology.orchestra.run.smithi078.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-15T20:26:55.389 INFO:teuthology.orchestra.run.smithi078.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-15T20:26:55.390 INFO:teuthology.orchestra.run.smithi146.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-15T20:26:55.390 INFO:teuthology.orchestra.run.smithi078.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-15T20:26:55.391 INFO:teuthology.orchestra.run.smithi146.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-15T20:26:55.448 INFO:teuthology.orchestra.run.smithi078.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-15T20:26:55.449 INFO:teuthology.orchestra.run.smithi146.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-15T20:26:55.450 INFO:teuthology.orchestra.run.smithi078.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-15T20:26:55.454 INFO:teuthology.orchestra.run.smithi078.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-15T20:26:55.456 INFO:teuthology.orchestra.run.smithi078.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-15T20:26:55.474 INFO:teuthology.orchestra.run.smithi146.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-15T20:26:55.475 INFO:teuthology.orchestra.run.smithi146.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-15T20:26:55.476 INFO:teuthology.orchestra.run.smithi146.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-15T20:26:55.486 INFO:teuthology.orchestra.run.smithi146.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-15T20:26:55.487 INFO:teuthology.orchestra.run.smithi146.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-15T20:26:55.488 INFO:teuthology.orchestra.run.smithi146.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-15T20:26:55.489 INFO:teuthology.orchestra.run.smithi146.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-15T20:26:55.529 INFO:teuthology.orchestra.run.smithi078.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-15T20:26:55.531 INFO:teuthology.orchestra.run.smithi078.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-15T20:26:55.532 INFO:teuthology.orchestra.run.smithi146.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-15T20:26:55.536 INFO:teuthology.orchestra.run.smithi146.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-15T20:26:55.537 INFO:teuthology.orchestra.run.smithi078.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-15T20:26:55.539 INFO:teuthology.orchestra.run.smithi078.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-15T20:26:55.541 INFO:teuthology.orchestra.run.smithi078.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-15T20:26:55.611 INFO:teuthology.orchestra.run.smithi078.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-15T20:26:55.612 INFO:teuthology.orchestra.run.smithi146.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-15T20:26:55.614 INFO:teuthology.orchestra.run.smithi146.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-15T20:26:55.616 INFO:teuthology.orchestra.run.smithi146.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-15T20:26:55.621 INFO:teuthology.orchestra.run.smithi146.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-15T20:26:55.622 INFO:teuthology.orchestra.run.smithi146.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-15T20:26:55.623 INFO:teuthology.orchestra.run.smithi146.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-15T20:26:55.628 INFO:teuthology.orchestra.run.smithi146.stdout:Get:63 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-15T20:26:55.729 INFO:teuthology.orchestra.run.smithi078.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-15T20:26:55.775 INFO:teuthology.orchestra.run.smithi078.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-15T20:26:55.777 INFO:teuthology.orchestra.run.smithi078.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T20:26:55.778 INFO:teuthology.orchestra.run.smithi078.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-15T20:26:55.780 INFO:teuthology.orchestra.run.smithi078.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-15T20:26:55.787 INFO:teuthology.orchestra.run.smithi146.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-15T20:26:55.789 INFO:teuthology.orchestra.run.smithi146.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-15T20:26:55.791 INFO:teuthology.orchestra.run.smithi146.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-15T20:26:55.792 INFO:teuthology.orchestra.run.smithi146.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-15T20:26:55.794 INFO:teuthology.orchestra.run.smithi146.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-15T20:26:55.820 INFO:teuthology.orchestra.run.smithi078.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-15T20:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-15T20:26:55.823 INFO:teuthology.orchestra.run.smithi078.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-15T20:26:55.860 INFO:teuthology.orchestra.run.smithi146.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-15T20:26:55.863 INFO:teuthology.orchestra.run.smithi146.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-15T20:26:55.863 INFO:teuthology.orchestra.run.smithi146.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-15T20:26:55.881 INFO:teuthology.orchestra.run.smithi078.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-15T20:26:55.882 INFO:teuthology.orchestra.run.smithi078.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-15T20:26:55.891 INFO:teuthology.orchestra.run.smithi146.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-15T20:26:55.891 INFO:teuthology.orchestra.run.smithi146.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-15T20:26:55.937 INFO:teuthology.orchestra.run.smithi078.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-15T20:26:55.938 INFO:teuthology.orchestra.run.smithi078.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-15T20:26:55.939 INFO:teuthology.orchestra.run.smithi078.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-15T20:26:55.941 INFO:teuthology.orchestra.run.smithi078.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-15T20:26:55.942 INFO:teuthology.orchestra.run.smithi078.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-15T20:26:55.943 INFO:teuthology.orchestra.run.smithi078.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-15T20:26:55.950 INFO:teuthology.orchestra.run.smithi078.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-15T20:26:55.954 INFO:teuthology.orchestra.run.smithi146.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-15T20:26:55.955 INFO:teuthology.orchestra.run.smithi146.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-15T20:26:55.956 INFO:teuthology.orchestra.run.smithi146.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-15T20:26:55.957 INFO:teuthology.orchestra.run.smithi146.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-15T20:26:55.957 INFO:teuthology.orchestra.run.smithi146.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-15T20:26:55.958 INFO:teuthology.orchestra.run.smithi146.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-15T20:26:55.964 INFO:teuthology.orchestra.run.smithi146.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-15T20:26:56.018 INFO:teuthology.orchestra.run.smithi078.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-15T20:26:56.021 INFO:teuthology.orchestra.run.smithi078.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-15T20:26:56.023 INFO:teuthology.orchestra.run.smithi078.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-15T20:26:56.035 INFO:teuthology.orchestra.run.smithi146.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-15T20:26:56.039 INFO:teuthology.orchestra.run.smithi146.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-15T20:26:56.040 INFO:teuthology.orchestra.run.smithi146.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-15T20:26:56.100 INFO:teuthology.orchestra.run.smithi078.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-15T20:26:56.106 INFO:teuthology.orchestra.run.smithi078.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-15T20:26:56.108 INFO:teuthology.orchestra.run.smithi078.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-15T20:26:56.110 INFO:teuthology.orchestra.run.smithi078.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-15T20:26:56.117 INFO:teuthology.orchestra.run.smithi146.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-15T20:26:56.121 INFO:teuthology.orchestra.run.smithi078.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-15T20:26:56.122 INFO:teuthology.orchestra.run.smithi078.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-15T20:26:56.124 INFO:teuthology.orchestra.run.smithi146.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-15T20:26:56.126 INFO:teuthology.orchestra.run.smithi146.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-15T20:26:56.128 INFO:teuthology.orchestra.run.smithi146.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2024-09-15T20:26:56.130 INFO:teuthology.orchestra.run.smithi078.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-15T20:26:56.139 INFO:teuthology.orchestra.run.smithi146.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-15T20:26:56.140 INFO:teuthology.orchestra.run.smithi146.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-15T20:26:56.148 INFO:teuthology.orchestra.run.smithi146.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-15T20:26:56.181 INFO:teuthology.orchestra.run.smithi078.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T20:26:56.183 INFO:teuthology.orchestra.run.smithi078.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-15T20:26:56.184 INFO:teuthology.orchestra.run.smithi078.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-15T20:26:56.199 INFO:teuthology.orchestra.run.smithi146.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-15T20:26:56.200 INFO:teuthology.orchestra.run.smithi146.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-15T20:26:56.201 INFO:teuthology.orchestra.run.smithi146.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-15T20:26:56.263 INFO:teuthology.orchestra.run.smithi078.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-15T20:26:56.265 INFO:teuthology.orchestra.run.smithi078.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-15T20:26:56.280 INFO:teuthology.orchestra.run.smithi146.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-15T20:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-15T20:26:56.282 INFO:teuthology.orchestra.run.smithi146.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-15T20:26:56.282 INFO:teuthology.orchestra.run.smithi078.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-15T20:26:56.283 INFO:teuthology.orchestra.run.smithi078.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-15T20:26:56.288 INFO:teuthology.orchestra.run.smithi078.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-15T20:26:56.290 INFO:teuthology.orchestra.run.smithi078.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-15T20:26:56.299 INFO:teuthology.orchestra.run.smithi146.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-15T20:26:56.300 INFO:teuthology.orchestra.run.smithi146.stdout:Get:97 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-15T20:26:56.300 INFO:teuthology.orchestra.run.smithi146.stdout:Get:98 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-15T20:26:56.306 INFO:teuthology.orchestra.run.smithi146.stdout:Get:99 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-15T20:26:56.307 INFO:teuthology.orchestra.run.smithi146.stdout:Get:100 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-15T20:26:56.802 INFO:teuthology.orchestra.run.smithi146.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-15T20:26:57.105 INFO:teuthology.orchestra.run.smithi146.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-15T20:26:57.110 INFO:teuthology.orchestra.run.smithi146.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-15T20:26:57.112 INFO:teuthology.orchestra.run.smithi146.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-15T20:26:57.257 INFO:teuthology.orchestra.run.smithi146.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-15T20:26:57.940 INFO:teuthology.orchestra.run.smithi146.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-15T20:26:58.433 INFO:teuthology.orchestra.run.smithi078.stdout:Get:101 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.3.0-4930-gfc3e0bff-1jammy [5,206 kB] 2024-09-15T20:26:59.258 INFO:teuthology.orchestra.run.smithi078.stdout:Get:102 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.3.0-4930-gfc3e0bff-1jammy [251 kB] 2024-09-15T20:26:59.329 INFO:teuthology.orchestra.run.smithi078.stdout:Get:103 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [124 kB] 2024-09-15T20:26:59.331 INFO:teuthology.orchestra.run.smithi078.stdout:Get:104 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.3.0-4930-gfc3e0bff-1jammy [1,561 kB] 2024-09-15T20:26:59.639 INFO:teuthology.orchestra.run.smithi078.stdout:Get:105 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.3.0-4930-gfc3e0bff-1jammy [6,309 kB] 2024-09-15T20:27:00.863 INFO:teuthology.orchestra.run.smithi078.stdout:Get:106 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.3.0-4930-gfc3e0bff-1jammy [23.1 MB] 2024-09-15T20:27:00.898 INFO:teuthology.orchestra.run.smithi146.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-15T20:27:00.899 INFO:teuthology.orchestra.run.smithi146.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-15T20:27:00.915 INFO:teuthology.orchestra.run.smithi146.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-15T20:27:01.267 INFO:teuthology.orchestra.run.smithi146.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-15T20:27:01.386 INFO:teuthology.orchestra.run.smithi146.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-15T20:27:01.415 INFO:teuthology.orchestra.run.smithi146.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-15T20:27:08.952 INFO:teuthology.orchestra.run.smithi078.stdout:Get:107 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.3.0-4930-gfc3e0bff-1jammy [14.1 kB] 2024-09-15T20:27:08.953 INFO:teuthology.orchestra.run.smithi078.stdout:Get:108 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [880 kB] 2024-09-15T20:27:09.527 INFO:teuthology.orchestra.run.smithi078.stdout:Get:109 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.3.0-4930-gfc3e0bff-1jammy [2,452 kB] 2024-09-15T20:27:11.552 INFO:teuthology.orchestra.run.smithi078.stdout:Get:110 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.3.0-4930-gfc3e0bff-1jammy [813 kB] 2024-09-15T20:27:12.184 INFO:teuthology.orchestra.run.smithi078.stdout:Get:111 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.3.0-4930-gfc3e0bff-1jammy [160 kB] 2024-09-15T20:27:12.305 INFO:teuthology.orchestra.run.smithi078.stdout:Get:112 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.3.0-4930-gfc3e0bff-1jammy [61.4 MB] 2024-09-15T20:27:15.633 INFO:teuthology.orchestra.run.smithi146.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-15T20:27:17.817 INFO:teuthology.orchestra.run.smithi146.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-15T20:27:17.874 INFO:teuthology.orchestra.run.smithi146.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-15T20:27:28.102 INFO:teuthology.orchestra.run.smithi146.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-15T20:27:28.105 INFO:teuthology.orchestra.run.smithi146.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-15T20:27:28.106 INFO:teuthology.orchestra.run.smithi146.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-15T20:27:31.252 INFO:teuthology.orchestra.run.smithi146.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-15T20:27:31.823 INFO:teuthology.orchestra.run.smithi146.stdout:Fetched 244 MB in 40s (6,143 kB/s) 2024-09-15T20:27:31.982 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-15T20:27:32.082 INFO:teuthology.orchestra.run.smithi146.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-15T20:27:32.088 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-15T20:27:32.172 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-15T20:27:32.467 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T20:27:32.493 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-15T20:27:32.525 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-15T20:27:33.028 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T20:27:33.054 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-15T20:27:33.095 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-15T20:27:33.482 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T20:27:33.508 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:27:33.565 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:27:33.935 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T20:27:33.961 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:27:34.001 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:27:34.337 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T20:27:34.364 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:27:34.404 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:27:34.732 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-15T20:27:34.758 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-15T20:27:34.798 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-15T20:27:35.105 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librados2. 2024-09-15T20:27:35.127 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:35.168 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:35.646 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libnbd0. 2024-09-15T20:27:35.672 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-15T20:27:35.712 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-15T20:27:36.015 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librbd1. 2024-09-15T20:27:36.041 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:36.081 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:36.509 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T20:27:36.535 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:36.575 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:36.870 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rados. 2024-09-15T20:27:36.885 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:36.928 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:37.222 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T20:27:37.248 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:27:37.289 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:37.516 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T20:27:37.542 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:37.583 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:37.886 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T20:27:37.912 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:27:37.952 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:38.438 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T20:27:38.464 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-15T20:27:38.505 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-15T20:27:38.824 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T20:27:38.851 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-15T20:27:38.891 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-15T20:27:39.110 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T20:27:39.125 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:39.160 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:39.496 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T20:27:39.522 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-15T20:27:39.563 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-15T20:27:39.840 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-15T20:27:39.855 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-15T20:27:39.898 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-15T20:27:40.201 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-15T20:27:40.228 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-15T20:27:40.268 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-15T20:27:40.529 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua5.1. 2024-09-15T20:27:40.544 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-15T20:27:40.588 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-15T20:27:40.907 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-any. 2024-09-15T20:27:40.922 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-15T20:27:40.966 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-15T20:27:41.210 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package zip. 2024-09-15T20:27:41.225 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-15T20:27:41.268 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking zip (3.0-12build2) ... 2024-09-15T20:27:41.588 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package unzip. 2024-09-15T20:27:41.614 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-15T20:27:41.654 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-15T20:27:42.008 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package luarocks. 2024-09-15T20:27:42.034 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-15T20:27:42.074 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-15T20:27:42.457 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package librgw2. 2024-09-15T20:27:42.483 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:42.523 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:43.118 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T20:27:43.137 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:43.317 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:43.645 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-15T20:27:43.672 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-15T20:27:43.712 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-15T20:27:44.240 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T20:27:44.266 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:44.306 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:44.651 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-common. 2024-09-15T20:27:44.677 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:44.712 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:46.173 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-base. 2024-09-15T20:27:46.199 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:46.249 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:46.827 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-15T20:27:46.854 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-15T20:27:46.893 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-15T20:27:47.154 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cheroot. 2024-09-15T20:27:47.169 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-15T20:27:47.204 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-15T20:27:47.490 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-15T20:27:47.506 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-15T20:27:47.549 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-15T20:27:47.843 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-15T20:27:47.870 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-15T20:27:47.910 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-15T20:27:48.212 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-15T20:27:48.239 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-15T20:27:48.304 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-15T20:27:48.590 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempora. 2024-09-15T20:27:48.605 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-15T20:27:48.639 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-15T20:27:48.901 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-portend. 2024-09-15T20:27:48.927 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-15T20:27:48.968 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-15T20:27:49.270 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-15T20:27:49.297 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-15T20:27:49.337 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-15T20:27:49.640 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T20:27:49.666 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-15T20:27:49.706 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-15T20:27:50.052 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T20:27:50.079 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-15T20:27:50.118 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-15T20:27:50.446 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T20:27:50.473 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-15T20:27:50.512 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-15T20:27:50.815 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T20:27:50.841 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-15T20:27:50.882 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-15T20:27:51.135 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-mako. 2024-09-15T20:27:51.161 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-15T20:27:51.193 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-15T20:27:51.488 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T20:27:51.514 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T20:27:51.554 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T20:27:51.873 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T20:27:51.900 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-15T20:27:51.940 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-15T20:27:52.243 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webob. 2024-09-15T20:27:52.269 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-15T20:27:52.309 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-15T20:27:52.654 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T20:27:52.680 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-15T20:27:52.724 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-15T20:27:52.998 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T20:27:53.024 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-15T20:27:53.056 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-15T20:27:53.351 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-paste. 2024-09-15T20:27:53.378 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-15T20:27:53.417 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-15T20:27:53.896 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T20:27:53.923 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-15T20:27:53.962 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-15T20:27:54.240 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T20:27:54.266 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-15T20:27:54.306 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-15T20:27:54.618 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T20:27:54.644 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-15T20:27:54.684 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-15T20:27:55.012 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T20:27:55.038 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-15T20:27:55.078 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-15T20:27:55.458 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-certifi. 2024-09-15T20:27:55.485 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-15T20:27:55.524 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-15T20:27:55.819 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-idna. 2024-09-15T20:27:55.834 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-15T20:27:55.869 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-15T20:27:56.172 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-urllib3. 2024-09-15T20:27:56.198 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-15T20:27:56.238 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-15T20:27:56.559 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-requests. 2024-09-15T20:27:56.584 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-15T20:27:56.624 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-15T20:27:56.936 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T20:27:56.962 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-15T20:27:57.002 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-15T20:27:57.297 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T20:27:57.312 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:27:57.355 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:57.718 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T20:27:57.745 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:57.776 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:58.054 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T20:27:58.081 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-15T20:27:58.120 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-15T20:27:58.448 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T20:27:58.475 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-15T20:27:58.540 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-15T20:27:58.885 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T20:27:58.911 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:58.951 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:59.329 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T20:27:59.356 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:27:59.387 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:27:59.966 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T20:27:59.992 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:00.032 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:01.052 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph. 2024-09-15T20:28:01.067 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:01.102 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:01.404 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T20:28:01.430 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:01.470 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:01.857 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T20:28:01.883 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:01.923 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:02.285 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package cephadm. 2024-09-15T20:28:02.311 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:02.351 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:02.645 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-15T20:28:02.672 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-15T20:28:02.712 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-15T20:28:02.981 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-15T20:28:02.996 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-15T20:28:03.031 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-15T20:28:03.980 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-babel. 2024-09-15T20:28:04.006 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-15T20:28:04.046 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-15T20:28:04.616 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T20:28:04.642 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-15T20:28:04.682 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-15T20:28:05.027 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T20:28:05.053 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:28:05.093 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:05.430 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T20:28:05.457 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T20:28:05.496 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T20:28:05.816 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-routes. 2024-09-15T20:28:05.842 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-15T20:28:05.882 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-15T20:28:06.244 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T20:28:06.271 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:28:06.310 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:08.931 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-15T20:28:08.948 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-15T20:28:08.989 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-15T20:28:09.468 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T20:28:09.495 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-15T20:28:09.543 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-15T20:28:09.913 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-15T20:28:09.941 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-15T20:28:09.979 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-15T20:28:10.291 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T20:28:10.318 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-15T20:28:10.442 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-15T20:28:11.118 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T20:28:11.146 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:28:11.184 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:12.138 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T20:28:12.166 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-15T20:28:12.204 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-15T20:28:12.482 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T20:28:12.510 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-15T20:28:12.540 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-15T20:28:12.835 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T20:28:12.863 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-15T20:28:12.910 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-15T20:28:13.197 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T20:28:13.224 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-15T20:28:13.263 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-15T20:28:13.583 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T20:28:13.611 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-15T20:28:13.649 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-15T20:28:13.961 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T20:28:13.989 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-15T20:28:14.028 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-15T20:28:14.330 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T20:28:14.346 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-15T20:28:14.389 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-15T20:28:14.909 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T20:28:14.936 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-15T20:28:14.975 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-15T20:28:15.605 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T20:28:15.622 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:28:15.663 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:15.925 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T20:28:15.952 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-15T20:28:15.991 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-15T20:28:16.344 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T20:28:16.372 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-15T20:28:16.411 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-15T20:28:16.671 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package jq. 2024-09-15T20:28:16.687 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-15T20:28:16.746 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-15T20:28:16.990 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package socat. 2024-09-15T20:28:17.007 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-15T20:28:17.040 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-15T20:28:17.310 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T20:28:17.326 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-15T20:28:17.369 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-15T20:28:17.723 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-test. 2024-09-15T20:28:17.751 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:17.789 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:19.951 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T20:28:19.967 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:28:20.001 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:20.305 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T20:28:20.322 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:20.355 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:20.650 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-15T20:28:20.678 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-15T20:28:20.716 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-15T20:28:21.046 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-15T20:28:21.074 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-15T20:28:21.112 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-15T20:28:21.475 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T20:28:21.502 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-15T20:28:21.566 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-15T20:28:21.980 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package pkg-config. 2024-09-15T20:28:22.007 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-15T20:28:22.037 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-15T20:28:22.282 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-15T20:28:22.310 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-15T20:28:22.348 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-15T20:28:22.743 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-15T20:28:22.759 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-15T20:28:22.801 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-15T20:28:23.104 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T20:28:23.132 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T20:28:23.171 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T20:28:23.491 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T20:28:23.518 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-15T20:28:23.557 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-15T20:28:23.868 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T20:28:23.896 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-15T20:28:23.935 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-15T20:28:24.271 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-py. 2024-09-15T20:28:24.299 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-15T20:28:24.337 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-15T20:28:24.649 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pygments. 2024-09-15T20:28:24.677 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-15T20:28:24.716 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-15T20:28:25.503 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T20:28:25.531 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-15T20:28:25.569 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-15T20:28:25.864 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-toml. 2024-09-15T20:28:25.892 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-15T20:28:25.930 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-15T20:28:26.183 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T20:28:26.211 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-15T20:28:26.249 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-15T20:28:26.603 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T20:28:26.631 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-15T20:28:26.669 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-15T20:28:26.939 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T20:28:26.955 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-15T20:28:26.997 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-15T20:28:27.726 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package radosgw. 2024-09-15T20:28:27.754 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:27.784 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:28.662 INFO:teuthology.orchestra.run.smithi146.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T20:28:28.690 INFO:teuthology.orchestra.run.smithi146.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:28:28.728 INFO:teuthology.orchestra.run.smithi146.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:29.025 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-15T20:28:29.266 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-15T20:28:29.384 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-15T20:28:29.641 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-15T20:28:29.957 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-15T20:28:30.365 INFO:teuthology.orchestra.run.smithi146.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-15T20:28:30.436 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:30.952 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user cephadm....done 2024-09-15T20:28:31.067 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-15T20:28:31.352 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-15T20:28:31.612 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-15T20:28:31.738 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-15T20:28:31.997 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T20:28:32.265 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-15T20:28:32.366 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-15T20:28:32.651 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-15T20:28:33.011 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-15T20:28:33.279 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-15T20:28:33.547 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-15T20:28:33.807 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-15T20:28:34.041 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-15T20:28:34.275 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-15T20:28:34.526 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:34.786 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-15T20:28:34.937 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-15T20:28:35.063 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-15T20:28:35.188 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-15T20:28:35.315 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-15T20:28:35.574 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-15T20:28:35.958 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-15T20:28:35.989 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-15T20:28:36.061 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-15T20:28:36.187 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-15T20:28:36.546 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-15T20:28:36.672 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-15T20:28:36.709 INFO:teuthology.orchestra.run.smithi078.stdout:Get:113 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.3.0-4930-gfc3e0bff-1jammy [8,625 kB] 2024-09-15T20:28:36.931 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-15T20:28:37.174 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-15T20:28:37.458 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up zip (3.0-12build2) ... 2024-09-15T20:28:37.601 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-15T20:28:38.295 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-15T20:28:38.554 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-15T20:28:38.680 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-15T20:28:38.798 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-15T20:28:39.082 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-15T20:28:39.450 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-15T20:28:39.701 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-15T20:28:40.094 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-15T20:28:40.387 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-15T20:28:40.731 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-15T20:28:40.973 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-15T20:28:41.082 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:41.367 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-15T20:28:41.693 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-15T20:28:42.865 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-15T20:28:43.100 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:28:43.234 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-15T20:28:43.494 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-15T20:28:43.620 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-15T20:28:43.746 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-15T20:28:44.047 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-15T20:28:44.323 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-15T20:28:44.558 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:28:44.684 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-15T20:28:44.976 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-15T20:28:45.238 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-15T20:28:45.506 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-15T20:28:45.897 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T20:28:46.133 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-15T20:28:46.392 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-15T20:28:46.518 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-15T20:28:46.632 INFO:teuthology.orchestra.run.smithi078.stdout:Get:114 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.3.0-4930-gfc3e0bff-1jammy [14.2 kB] 2024-09-15T20:28:46.662 INFO:teuthology.orchestra.run.smithi078.stdout:Get:115 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.3.0-4930-gfc3e0bff-1jammy [48.5 MB] 2024-09-15T20:28:46.794 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-15T20:28:46.920 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-15T20:28:47.230 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-15T20:28:47.539 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-15T20:28:47.782 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-15T20:28:47.899 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-15T20:28:48.025 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-15T20:28:48.115 INFO:teuthology.orchestra.run.smithi146.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-15T20:28:48.361 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-15T20:28:48.486 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-15T20:28:48.888 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-15T20:28:49.156 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-15T20:28:49.265 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-15T20:28:49.499 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:28:49.625 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-15T20:28:49.935 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-15T20:28:50.244 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-15T20:28:50.529 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-15T20:28:50.655 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-15T20:28:50.931 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-15T20:28:51.259 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-15T20:28:51.560 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-15T20:28:51.937 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T20:28:52.238 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-15T20:28:52.566 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-15T20:28:52.683 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:52.809 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:52.935 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-15T20:28:53.280 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:53.406 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:53.524 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-15T20:28:53.775 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:53.901 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:54.027 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:54.239 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T20:28:54.239 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T20:28:54.703 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:54.846 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:54.964 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:55.090 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-15T20:28:56.227 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:56.353 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:56.479 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:56.605 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:56.731 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:57.240 INFO:teuthology.orchestra.run.smithi146.stdout:Adding group ceph....done 2024-09-15T20:28:57.599 INFO:teuthology.orchestra.run.smithi146.stdout:Adding system user ceph....done 2024-09-15T20:28:57.726 INFO:teuthology.orchestra.run.smithi146.stdout:Setting system user ceph properties....done 2024-09-15T20:28:57.745 INFO:teuthology.orchestra.run.smithi146.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T20:28:57.880 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T20:28:58.215 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T20:28:58.657 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:58.766 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:59.216 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:28:59.216 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:28:59.643 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:28:59.986 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T20:29:00.487 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T20:29:00.830 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T20:29:01.297 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T20:29:01.498 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T20:29:01.981 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:02.256 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T20:29:02.256 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T20:29:02.733 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:02.859 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:03.027 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:03.241 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T20:29:03.241 INFO:teuthology.orchestra.run.smithi146.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T20:29:03.761 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:03.921 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:04.030 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:04.164 INFO:teuthology.orchestra.run.smithi146.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:04.501 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-15T20:29:04.601 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-15T20:29:04.753 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-15T20:29:05.829 INFO:teuthology.orchestra.run.smithi146.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-15T20:29:06.498 INFO:teuthology.orchestra.run.smithi146.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T20:29:06.504 DEBUG:teuthology.parallel:result is None 2024-09-15T20:29:15.131 INFO:teuthology.orchestra.run.smithi078.stdout:Get:116 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.3.0-4930-gfc3e0bff-1jammy [130 kB] 2024-09-15T20:29:15.133 INFO:teuthology.orchestra.run.smithi078.stdout:Get:117 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.3.0-4930-gfc3e0bff-1jammy [39.0 kB] 2024-09-15T20:29:15.172 INFO:teuthology.orchestra.run.smithi078.stdout:Get:118 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.3.0-4930-gfc3e0bff-1jammy [15.8 MB] 2024-09-15T20:29:22.264 INFO:teuthology.orchestra.run.smithi078.stdout:Get:119 https://3.chacra.ceph.com/r/ceph/main/fc3e0bff7468fd55478adbae7f75508b749124b6/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.3.0-4930-gfc3e0bff-1jammy [91.2 kB] 2024-09-15T20:29:22.841 INFO:teuthology.orchestra.run.smithi078.stdout:Fetched 244 MB in 2min 31s (1,616 kB/s) 2024-09-15T20:29:23.007 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-15T20:29:23.106 INFO:teuthology.orchestra.run.smithi078.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165990 files and directories currently installed.) 2024-09-15T20:29:23.112 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-15T20:29:23.197 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-15T20:29:23.492 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-15T20:29:23.519 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-15T20:29:23.559 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-15T20:29:23.879 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-15T20:29:23.905 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-15T20:29:23.945 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-15T20:29:24.407 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-15T20:29:24.434 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:29:24.482 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:29:24.844 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-15T20:29:24.871 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:29:24.902 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:29:25.247 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-15T20:29:25.274 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-15T20:29:25.313 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:29:25.650 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-15T20:29:25.677 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-15T20:29:25.717 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-15T20:29:26.020 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librados2. 2024-09-15T20:29:26.046 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../007-librados2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:26.086 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:26.581 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libnbd0. 2024-09-15T20:29:26.607 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../008-libnbd0_1.10.5-1_amd64.deb ... 2024-09-15T20:29:26.647 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libnbd0 (1.10.5-1) ... 2024-09-15T20:29:26.917 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librbd1. 2024-09-15T20:29:26.943 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../009-librbd1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:26.975 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:27.412 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs2. 2024-09-15T20:29:27.438 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../010-libcephfs2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:27.478 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:27.756 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rados. 2024-09-15T20:29:27.782 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../011-python3-rados_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:27.822 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:28.159 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-15T20:29:28.185 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../012-python3-ceph-argparse_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:29:28.217 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:28.436 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cephfs. 2024-09-15T20:29:28.463 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../013-python3-cephfs_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:28.503 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:28.806 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-15T20:29:28.832 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../014-python3-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:29:28.873 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:29.176 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-15T20:29:29.202 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../015-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-15T20:29:29.244 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-15T20:29:29.554 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-prettytable. 2024-09-15T20:29:29.580 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../016-python3-prettytable_2.5.0-2_all.deb ... 2024-09-15T20:29:29.620 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-15T20:29:29.881 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rbd. 2024-09-15T20:29:29.896 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../017-python3-rbd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:29.940 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:30.268 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-15T20:29:30.294 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../018-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-15T20:29:30.326 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-15T20:29:30.595 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-15T20:29:30.611 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../019-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-15T20:29:30.653 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-15T20:29:31.040 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-15T20:29:31.066 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../020-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-15T20:29:31.165 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-15T20:29:31.544 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua5.1. 2024-09-15T20:29:31.557 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../021-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-15T20:29:31.602 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-15T20:29:31.914 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-any. 2024-09-15T20:29:31.929 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../022-lua-any_27ubuntu1_all.deb ... 2024-09-15T20:29:31.972 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-15T20:29:32.208 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package zip. 2024-09-15T20:29:32.223 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../023-zip_3.0-12build2_amd64.deb ... 2024-09-15T20:29:32.266 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking zip (3.0-12build2) ... 2024-09-15T20:29:32.586 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package unzip. 2024-09-15T20:29:32.613 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../024-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-15T20:29:32.653 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-15T20:29:33.049 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package luarocks. 2024-09-15T20:29:33.075 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../025-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-15T20:29:33.115 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-15T20:29:33.490 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package librgw2. 2024-09-15T20:29:33.517 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../026-librgw2_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:33.556 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:34.159 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rgw. 2024-09-15T20:29:34.186 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../027-python3-rgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:34.234 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:34.545 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-15T20:29:34.572 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../028-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-15T20:29:34.612 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-15T20:29:34.898 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libradosstriper1. 2024-09-15T20:29:34.925 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../029-libradosstriper1_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:34.965 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:35.419 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-common. 2024-09-15T20:29:35.445 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../030-ceph-common_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:35.485 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:36.949 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-base. 2024-09-15T20:29:36.975 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../031-ceph-base_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:37.025 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:37.595 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-15T20:29:37.622 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../032-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-15T20:29:37.661 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-15T20:29:37.964 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cheroot. 2024-09-15T20:29:37.991 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../033-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-15T20:29:38.031 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-15T20:29:38.351 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-15T20:29:38.377 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../034-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-15T20:29:38.426 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-15T20:29:38.721 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-15T20:29:38.747 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../035-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-15T20:29:38.787 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-15T20:29:39.090 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-15T20:29:39.117 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../036-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-15T20:29:39.157 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-15T20:29:39.460 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tempora. 2024-09-15T20:29:39.486 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../037-python3-tempora_4.1.2-1_all.deb ... 2024-09-15T20:29:39.526 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-15T20:29:39.796 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-portend. 2024-09-15T20:29:39.823 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../038-python3-portend_3.0.0-1_all.deb ... 2024-09-15T20:29:39.854 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-15T20:29:40.157 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-15T20:29:40.183 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../039-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-15T20:29:40.224 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-15T20:29:40.510 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-15T20:29:40.536 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../040-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-15T20:29:40.577 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-15T20:29:40.914 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-natsort. 2024-09-15T20:29:40.940 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../041-python3-natsort_8.0.2-1_all.deb ... 2024-09-15T20:29:40.980 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-15T20:29:41.308 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-logutils. 2024-09-15T20:29:41.335 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../042-python3-logutils_0.3.3-8_all.deb ... 2024-09-15T20:29:41.375 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-15T20:29:41.670 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-15T20:29:41.696 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../043-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-15T20:29:41.736 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-15T20:29:42.048 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-mako. 2024-09-15T20:29:42.074 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../044-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-15T20:29:42.114 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-15T20:29:42.468 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-15T20:29:42.494 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../045-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-15T20:29:42.534 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-15T20:29:42.837 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-15T20:29:42.864 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../046-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-15T20:29:42.904 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-15T20:29:43.182 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webob. 2024-09-15T20:29:43.208 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../047-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-15T20:29:43.248 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-15T20:29:43.593 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-waitress. 2024-09-15T20:29:43.619 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../048-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-15T20:29:43.663 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-15T20:29:43.954 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-tempita. 2024-09-15T20:29:43.981 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../049-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-15T20:29:44.021 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-15T20:29:44.324 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-paste. 2024-09-15T20:29:44.351 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../050-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-15T20:29:44.390 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-15T20:29:44.719 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-15T20:29:44.734 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../051-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-15T20:29:44.777 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-15T20:29:45.072 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-15T20:29:45.098 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../052-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-15T20:29:45.138 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-15T20:29:45.441 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-webtest. 2024-09-15T20:29:45.467 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../053-python3-webtest_2.0.35-1_all.deb ... 2024-09-15T20:29:45.499 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-15T20:29:45.811 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pecan. 2024-09-15T20:29:45.837 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../054-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-15T20:29:45.877 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-15T20:29:46.224 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-certifi. 2024-09-15T20:29:46.251 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../055-python3-certifi_2020.6.20-1_all.deb ... 2024-09-15T20:29:46.291 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-15T20:29:46.610 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-idna. 2024-09-15T20:29:46.636 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../056-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-15T20:29:46.685 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-15T20:29:47.005 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-urllib3. 2024-09-15T20:29:47.031 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../057-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-15T20:29:47.071 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-15T20:29:47.400 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-requests. 2024-09-15T20:29:47.426 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../058-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-15T20:29:47.466 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-15T20:29:47.753 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-15T20:29:47.779 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../059-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-15T20:29:47.811 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-15T20:29:48.114 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-15T20:29:48.129 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../060-ceph-mgr-modules-core_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:29:48.172 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:48.570 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-15T20:29:48.596 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../061-libsqlite3-mod-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:48.636 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:48.947 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-15T20:29:48.974 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../062-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-15T20:29:49.014 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-15T20:29:49.308 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-openssl. 2024-09-15T20:29:49.335 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../063-python3-openssl_21.0.0-1_all.deb ... 2024-09-15T20:29:49.386 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-15T20:29:49.712 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr. 2024-09-15T20:29:49.738 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../064-ceph-mgr_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:49.778 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:50.173 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mon. 2024-09-15T20:29:50.196 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../065-ceph-mon_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:50.239 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:50.827 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-osd. 2024-09-15T20:29:50.853 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../066-ceph-osd_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:50.884 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:52.121 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph. 2024-09-15T20:29:52.161 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../067-ceph_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:52.179 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:52.499 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-fuse. 2024-09-15T20:29:52.525 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../068-ceph-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:52.565 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:52.918 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mds. 2024-09-15T20:29:52.945 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../069-ceph-mds_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:53.002 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:53.405 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package cephadm. 2024-09-15T20:29:53.431 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../070-cephadm_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:29:53.471 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:53.766 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-15T20:29:53.793 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../071-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-15T20:29:53.832 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-15T20:29:54.143 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-15T20:29:54.159 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../072-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-15T20:29:54.202 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-15T20:29:55.201 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-babel. 2024-09-15T20:29:55.228 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../073-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-15T20:29:55.267 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-15T20:29:55.587 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-jinja2. 2024-09-15T20:29:55.614 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../074-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-15T20:29:55.653 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-15T20:29:56.015 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-15T20:29:56.042 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../075-ceph-mgr-cephadm_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:29:56.081 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:56.435 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-15T20:29:56.462 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../076-python3-repoze.lru_0.7-2_all.deb ... 2024-09-15T20:29:56.502 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-15T20:29:56.796 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-routes. 2024-09-15T20:29:56.823 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../077-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-15T20:29:56.863 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-15T20:29:57.183 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-15T20:29:57.210 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../078-ceph-mgr-dashboard_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:29:57.241 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:29:59.972 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-15T20:30:00.001 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../079-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-15T20:30:00.055 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-15T20:30:00.558 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-joblib. 2024-09-15T20:30:00.586 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../080-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-15T20:30:00.616 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-15T20:30:00.995 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-15T20:30:01.022 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../081-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-15T20:30:01.061 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-15T20:30:01.373 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-sklearn. 2024-09-15T20:30:01.400 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../082-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-15T20:30:01.525 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-15T20:30:02.243 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-15T20:30:02.268 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../083-ceph-mgr-diskprediction-local_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:30:02.301 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:03.296 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-cachetools. 2024-09-15T20:30:03.324 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../084-python3-cachetools_5.0.0-1_all.deb ... 2024-09-15T20:30:03.363 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-15T20:30:03.666 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-15T20:30:03.693 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../085-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-15T20:30:03.757 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-15T20:30:04.086 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-15T20:30:04.113 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../086-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-15T20:30:04.152 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-15T20:30:04.472 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-rsa. 2024-09-15T20:30:04.501 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../087-python3-rsa_4.8-1_all.deb ... 2024-09-15T20:30:04.539 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-15T20:30:04.810 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-google-auth. 2024-09-15T20:30:04.838 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../088-python3-google-auth_1.5.1-3_all.deb ... 2024-09-15T20:30:04.876 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-15T20:30:05.196 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-15T20:30:05.224 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../089-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-15T20:30:05.262 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-15T20:30:05.574 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-websocket. 2024-09-15T20:30:05.602 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../090-python3-websocket_1.2.3-1_all.deb ... 2024-09-15T20:30:05.641 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-15T20:30:05.952 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-15T20:30:05.980 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../091-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-15T20:30:06.019 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-15T20:30:06.650 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-15T20:30:06.679 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../092-ceph-mgr-k8sevents_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:30:06.716 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:07.044 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-15T20:30:07.072 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../093-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-15T20:30:07.111 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-15T20:30:07.622 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-15T20:30:07.650 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../094-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-15T20:30:07.697 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-15T20:30:07.958 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package jq. 2024-09-15T20:30:07.974 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../095-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-15T20:30:08.016 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-15T20:30:08.269 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package socat. 2024-09-15T20:30:08.286 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../096-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-15T20:30:08.327 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-15T20:30:08.631 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package xmlstarlet. 2024-09-15T20:30:08.647 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../097-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-15T20:30:08.681 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-15T20:30:09.044 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-test. 2024-09-15T20:30:09.072 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../098-ceph-test_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:30:09.110 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:11.390 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package ceph-volume. 2024-09-15T20:30:11.418 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../099-ceph-volume_19.3.0-4930-gfc3e0bff-1jammy_all.deb ... 2024-09-15T20:30:11.456 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:11.744 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-15T20:30:11.760 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../100-libcephfs-dev_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:30:11.802 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:12.105 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-15T20:30:12.133 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../101-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-15T20:30:12.172 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-15T20:30:12.510 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-15T20:30:12.537 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../102-lua-sec_1.0.2-1_amd64.deb ... 2024-09-15T20:30:12.576 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-15T20:30:12.947 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package nvme-cli. 2024-09-15T20:30:12.974 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../103-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2024-09-15T20:30:13.013 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2024-09-15T20:30:13.435 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package pkg-config. 2024-09-15T20:30:13.463 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../104-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-15T20:30:13.493 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-15T20:30:13.738 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-15T20:30:13.766 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../105-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-15T20:30:13.804 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-15T20:30:14.192 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-15T20:30:14.219 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../106-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-15T20:30:14.258 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-15T20:30:14.553 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pastescript. 2024-09-15T20:30:14.580 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../107-python3-pastescript_2.0.2-4_all.deb ... 2024-09-15T20:30:14.620 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-15T20:30:14.897 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pluggy. 2024-09-15T20:30:14.914 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../108-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-15T20:30:14.956 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-15T20:30:15.276 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-psutil. 2024-09-15T20:30:15.304 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../109-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-15T20:30:15.342 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-15T20:30:15.679 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-py. 2024-09-15T20:30:15.706 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../110-python3-py_1.10.0-1_all.deb ... 2024-09-15T20:30:15.745 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-15T20:30:16.052 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pygments. 2024-09-15T20:30:16.077 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../111-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-15T20:30:16.115 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-15T20:30:16.586 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-15T20:30:16.614 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../112-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-15T20:30:16.652 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-15T20:30:16.947 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-toml. 2024-09-15T20:30:16.975 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../113-python3-toml_0.10.2-1_all.deb ... 2024-09-15T20:30:17.013 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-15T20:30:17.308 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-pytest. 2024-09-15T20:30:17.325 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../114-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-15T20:30:17.366 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-15T20:30:17.712 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package python3-simplejson. 2024-09-15T20:30:17.833 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../115-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-15T20:30:17.833 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-15T20:30:18.081 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-15T20:30:18.109 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../116-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-15T20:30:18.148 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-15T20:30:18.861 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package radosgw. 2024-09-15T20:30:18.889 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../117-radosgw_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:30:18.927 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:19.855 INFO:teuthology.orchestra.run.smithi078.stdout:Selecting previously unselected package rbd-fuse. 2024-09-15T20:30:19.871 INFO:teuthology.orchestra.run.smithi078.stdout:Preparing to unpack .../118-rbd-fuse_19.3.0-4930-gfc3e0bff-1jammy_amd64.deb ... 2024-09-15T20:30:19.913 INFO:teuthology.orchestra.run.smithi078.stdout:Unpacking rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:20.202 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-15T20:30:20.452 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-15T20:30:20.578 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2024-09-15T20:30:20.837 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-15T20:30:21.174 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-15T20:30:21.613 INFO:teuthology.orchestra.run.smithi078.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-15T20:30:21.688 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:22.213 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user cephadm....done 2024-09-15T20:30:22.361 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-15T20:30:22.621 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-15T20:30:23.139 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-15T20:30:23.265 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-15T20:30:23.516 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-15T20:30:23.776 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-15T20:30:23.877 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-15T20:30:24.187 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-15T20:30:24.539 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-15T20:30:24.807 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-15T20:30:25.075 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-15T20:30:25.402 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-15T20:30:25.636 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-15T20:30:25.896 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-15T20:30:26.155 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-argparse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:26.423 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-15T20:30:26.549 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libnbd0 (1.10.5-1) ... 2024-09-15T20:30:26.675 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-15T20:30:26.793 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-15T20:30:26.894 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-15T20:30:27.137 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-15T20:30:27.650 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-15T20:30:27.685 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-15T20:30:27.749 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-15T20:30:27.867 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-15T20:30:28.235 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-15T20:30:28.335 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-15T20:30:28.595 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-15T20:30:28.863 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-15T20:30:29.189 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up zip (3.0-12build2) ... 2024-09-15T20:30:29.324 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-15T20:30:30.026 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-15T20:30:30.252 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-15T20:30:30.370 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-15T20:30:30.496 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-15T20:30:30.789 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-15T20:30:31.165 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-15T20:30:31.416 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-15T20:30:31.826 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-15T20:30:32.094 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-15T20:30:32.429 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-15T20:30:32.680 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-15T20:30:32.806 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:33.116 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-15T20:30:33.434 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-15T20:30:34.591 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-15T20:30:34.842 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:30:34.952 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-15T20:30:35.195 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-15T20:30:35.321 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-15T20:30:35.446 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-15T20:30:35.748 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-15T20:30:36.025 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-15T20:30:36.275 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:30:36.377 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-15T20:30:36.653 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-15T20:30:36.904 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-15T20:30:37.164 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-15T20:30:37.451 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-15T20:30:37.685 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-15T20:30:37.927 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-15T20:30:38.037 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-15T20:30:38.321 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-15T20:30:38.447 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-15T20:30:38.749 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-15T20:30:39.049 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-15T20:30:39.318 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-15T20:30:39.435 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-15T20:30:39.536 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-15T20:30:39.642 INFO:teuthology.orchestra.run.smithi078.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-15T20:30:39.879 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-15T20:30:40.005 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-15T20:30:40.398 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-15T20:30:40.666 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-15T20:30:40.792 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-15T20:30:41.051 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-15T20:30:41.153 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-15T20:30:41.470 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-15T20:30:41.763 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-15T20:30:42.031 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-15T20:30:42.157 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-15T20:30:42.424 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-15T20:30:42.786 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-15T20:30:43.037 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-15T20:30:43.413 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-15T20:30:43.698 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-15T20:30:44.033 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-15T20:30:44.185 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librados2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:44.294 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libsqlite3-mod-ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:44.403 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-15T20:30:44.739 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:44.865 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libradosstriper1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:44.991 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-15T20:30:45.259 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librbd1 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:45.360 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-modules-core (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:45.461 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:45.667 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T20:30:45.668 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-15T20:30:46.138 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up libcephfs-dev (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:46.289 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rados (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:46.432 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up librgw2 (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:46.591 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-15T20:30:47.795 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rbd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:47.921 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up rbd-fuse (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:48.047 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-rgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:48.173 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up python3-cephfs (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:48.299 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-common (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:49.150 INFO:teuthology.orchestra.run.smithi078.stdout:Adding group ceph....done 2024-09-15T20:30:49.535 INFO:teuthology.orchestra.run.smithi078.stdout:Adding system user ceph....done 2024-09-15T20:30:49.703 INFO:teuthology.orchestra.run.smithi078.stdout:Setting system user ceph properties....done 2024-09-15T20:30:49.721 INFO:teuthology.orchestra.run.smithi078.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-15T20:30:49.855 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-15T20:30:50.190 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-15T20:30:50.675 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-test (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:50.784 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up radosgw (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:51.203 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:30:51.204 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-15T20:30:51.695 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-base (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:52.041 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-15T20:30:52.513 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mds (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:52.744 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T20:30:52.744 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-15T20:30:53.198 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:53.406 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T20:30:53.407 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-15T20:30:54.053 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-osd (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:54.460 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T20:30:54.460 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-15T20:30:54.987 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-k8sevents (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:55.130 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-diskprediction-local (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:55.281 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mon (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:55.494 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T20:30:55.494 INFO:teuthology.orchestra.run.smithi078.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-15T20:30:55.949 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-cephadm (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:56.092 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:56.210 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-mgr-dashboard (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:56.336 INFO:teuthology.orchestra.run.smithi078.stdout:Setting up ceph-volume (19.3.0-4930-gfc3e0bff-1jammy) ... 2024-09-15T20:30:56.705 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-15T20:30:56.814 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-15T20:30:56.967 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-15T20:30:57.766 INFO:teuthology.orchestra.run.smithi078.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-15T20:30:58.445 INFO:teuthology.orchestra.run.smithi078.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-15T20:30:58.451 DEBUG:teuthology.parallel:result is None 2024-09-15T20:30:58.452 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:30:58.596 DEBUG:teuthology.orchestra.run.smithi078:> dpkg-query -W -f '${Version}' ceph 2024-09-15T20:30:58.615 INFO:teuthology.orchestra.run.smithi078.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:30:58.615 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:30:58.615 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-15T20:30:58.617 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=fc3e0bff7468fd55478adbae7f75508b749124b6 2024-09-15T20:30:58.754 DEBUG:teuthology.orchestra.run.smithi146:> dpkg-query -W -f '${Version}' ceph 2024-09-15T20:30:58.773 INFO:teuthology.orchestra.run.smithi146.stdout:19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:30:58.774 INFO:teuthology.packaging:The installed version of ceph is 19.3.0-4930-gfc3e0bff-1jammy 2024-09-15T20:30:58.774 INFO:teuthology.task.install:The correct ceph version 19.3.0-4930-gfc3e0bff-1jammy is installed. 2024-09-15T20:30:58.775 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-15T20:30:58.775 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:30:58.776 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T20:30:58.790 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:30:58.790 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-15T20:30:58.830 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-15T20:30:58.830 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:30:58.830 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T20:30:58.845 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T20:30:58.900 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:30:58.900 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/daemon-helper 2024-09-15T20:30:58.913 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-15T20:30:58.967 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-15T20:30:58.967 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:30:58.967 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T20:30:58.980 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T20:30:59.032 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:30:59.032 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-15T20:30:59.045 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-15T20:30:59.099 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-15T20:30:59.100 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:30:59.100 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T20:30:59.113 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T20:30:59.168 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:30:59.168 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/usr/bin/stdin-killer 2024-09-15T20:30:59.181 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-15T20:30:59.235 INFO:teuthology.run_tasks:Running task ceph... 2024-09-15T20:30:59.311 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-15T20:30:59.312 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 777 /var/log/ceph 2024-09-15T20:30:59.314 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 777 /var/log/ceph 2024-09-15T20:30:59.326 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-15T20:30:59.326 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T20:30:59.370 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-15T20:30:59.382 INFO:tasks.ceph:Creating extra log directories... 2024-09-15T20:30:59.382 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T20:30:59.426 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-15T20:30:59.447 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-09-15T20:30:59.447 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-09-15T20:30:59.448 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600', 'branch': 'main', 'description': 'rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_api_tests}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7905600', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1}, 'global': {'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'ScrubResult', 'scrub mismatch', 'overall HEALTH_', '\\(MGR_DOWN\\)', '\\(PG_AVAILABILITY\\)', '\\(SLOW_OPS\\)', 'slow request', '\\(OSD_SLOW_PING_TIME', 'overall HEALTH_', '\\(MON_DOWN\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_FULL\\)', '\\(SLOW_OPS\\)', '\\(MON_DOWN\\)', '\\(PG_', '\\(SMALLER_PGP_NUM\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '7498493279982dfe87f00616198a5967475f6169'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 101, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['mon.a', 'mon.c', 'osd.0', 'osd.1', 'osd.2'], ['mon.b', 'mgr.x', 'osd.3', 'osd.4', 'osd.5', 'client.0']], 'seed': 2855, 'sha1': 'fc3e0bff7468fd55478adbae7f75508b749124b6', 'sleep_before_teardown': 0, 'subset': '17936/100000', 'suite': 'rados', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_7498493279982dfe87f00616198a5967475f6169/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': '7498493279982dfe87f00616198a5967475f6169', 'targets': {'smithi078.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBDeyI8ErHjmywyAcgaaMxyLBjW6vQ/ZTkX96c5IAX6jwub0/UwN3ibsTL6P/F4ucTPWzc0diLCQKX1b5ouizXwA=', 'smithi146.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBXaqSChWCsjJib9s1kXCCHha6GFby9mTWPWasr8GTz3rx/YfyXumXWeqhTzNH4vaqHa7qSVZxuF7//bK3bqR6g='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'mon_thrash': {'revive_delay': 90, 'thrash_delay': 1}}, {'workunit': {'clients': {'client.0': ['rados/test.sh unit_test_scan']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-15_20:00:16', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-15T20:30:59.448 DEBUG:teuthology.orchestra.run.smithi078:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T20:30:59.478 DEBUG:teuthology.orchestra.run.smithi146:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-09-15T20:30:59.485 DEBUG:teuthology.orchestra.run.smithi078:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T20:30:59.526 DEBUG:teuthology.orchestra.run.smithi146:> sudo install -d -m0777 -- /var/run/ceph 2024-09-15T20:30:59.545 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:30:59.545 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T20:30:59.586 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T20:30:59.586 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_1 2024-09-15T20:30:59.634 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T20:30:59.634 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:30:59.634 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 878 Links: 1 2024-09-15T20:30:59.634 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:30:59.634 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-15 20:23:08.788478985 +0000 2024-09-15T20:30:59.634 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-15 20:23:08.528481308 +0000 2024-09-15T20:30:59.634 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-15 20:23:08.528481308 +0000 2024-09-15T20:30:59.635 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-09-15T20:30:59.635 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T20:30:59.688 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-15T20:30:59.688 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-15T20:30:59.688 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000440342 s, 1.2 MB/s 2024-09-15T20:30:59.690 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T20:30:59.739 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_2 2024-09-15T20:30:59.785 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T20:30:59.785 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:30:59.785 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 892 Links: 1 2024-09-15T20:30:59.786 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:30:59.786 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-15 20:23:09.288474520 +0000 2024-09-15T20:30:59.786 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-15 20:23:09.004477056 +0000 2024-09-15T20:30:59.786 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-15 20:23:09.004477056 +0000 2024-09-15T20:30:59.786 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-09-15T20:30:59.786 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T20:30:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-15T20:30:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-15T20:30:59.839 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000397326 s, 1.3 MB/s 2024-09-15T20:30:59.841 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T20:30:59.890 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_3 2024-09-15T20:30:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T20:30:59.937 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:30:59.937 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 916 Links: 1 2024-09-15T20:30:59.938 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:30:59.938 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-15 20:23:09.772470197 +0000 2024-09-15T20:30:59.938 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-15 20:23:09.516472483 +0000 2024-09-15T20:30:59.938 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-15 20:23:09.516472483 +0000 2024-09-15T20:30:59.938 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-09-15T20:30:59.938 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T20:30:59.991 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-15T20:30:59.991 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-15T20:30:59.991 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000434792 s, 1.2 MB/s 2024-09-15T20:30:59.993 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T20:31:00.043 DEBUG:teuthology.orchestra.run.smithi078:> stat /dev/vg_nvme/lv_4 2024-09-15T20:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T20:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout:Device: 5h/5d Inode: 927 Links: 1 2024-09-15T20:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout:Access: 2024-09-15 20:23:10.016468016 +0000 2024-09-15T20:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout:Modify: 2024-09-15 20:23:10.012468052 +0000 2024-09-15T20:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout:Change: 2024-09-15 20:23:10.012468052 +0000 2024-09-15T20:31:00.090 INFO:teuthology.orchestra.run.smithi078.stdout: Birth: - 2024-09-15T20:31:00.090 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T20:31:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records in 2024-09-15T20:31:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:1+0 records out 2024-09-15T20:31:00.144 INFO:teuthology.orchestra.run.smithi078.stderr:512 bytes copied, 0.000363652 s, 1.4 MB/s 2024-09-15T20:31:00.145 DEBUG:teuthology.orchestra.run.smithi078:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T20:31:00.195 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T20:31:00.195 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:00.195 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/scratch_devs of=/dev/stdout 2024-09-15T20:31:00.201 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-15T20:31:00.201 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_1 2024-09-15T20:31:00.249 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-15T20:31:00.249 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:31:00.249 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 882 Links: 1 2024-09-15T20:31:00.249 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:31:00.249 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-15 20:23:08.974795679 +0000 2024-09-15T20:31:00.249 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-15 20:23:08.970795716 +0000 2024-09-15T20:31:00.250 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-15 20:23:08.970795716 +0000 2024-09-15T20:31:00.250 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-09-15T20:31:00.250 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-15T20:31:00.304 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-15T20:31:00.304 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-15T20:31:00.304 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000430884 s, 1.2 MB/s 2024-09-15T20:31:00.305 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-15T20:31:00.354 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_2 2024-09-15T20:31:00.401 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-15T20:31:00.401 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:31:00.401 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 899 Links: 1 2024-09-15T20:31:00.401 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:31:00.401 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-15 20:23:09.750788749 +0000 2024-09-15T20:31:00.401 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-15 20:23:09.474791214 +0000 2024-09-15T20:31:00.401 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-15 20:23:09.474791214 +0000 2024-09-15T20:31:00.401 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-09-15T20:31:00.402 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-15T20:31:00.455 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-15T20:31:00.455 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-15T20:31:00.455 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.00035111 s, 1.5 MB/s 2024-09-15T20:31:00.456 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-15T20:31:00.506 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_3 2024-09-15T20:31:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-15T20:31:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:31:00.553 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 914 Links: 1 2024-09-15T20:31:00.553 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:31:00.553 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-15 20:23:10.286783961 +0000 2024-09-15T20:31:00.553 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-15 20:23:10.010786427 +0000 2024-09-15T20:31:00.553 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-15 20:23:10.010786427 +0000 2024-09-15T20:31:00.553 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-09-15T20:31:00.554 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-15T20:31:00.606 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-15T20:31:00.607 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-15T20:31:00.607 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000389093 s, 1.3 MB/s 2024-09-15T20:31:00.608 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-15T20:31:00.658 DEBUG:teuthology.orchestra.run.smithi146:> stat /dev/vg_nvme/lv_4 2024-09-15T20:31:00.704 INFO:teuthology.orchestra.run.smithi146.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-15T20:31:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-15T20:31:00.705 INFO:teuthology.orchestra.run.smithi146.stdout:Device: 5h/5d Inode: 933 Links: 1 2024-09-15T20:31:00.705 INFO:teuthology.orchestra.run.smithi146.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-15T20:31:00.705 INFO:teuthology.orchestra.run.smithi146.stdout:Access: 2024-09-15 20:23:10.766779673 +0000 2024-09-15T20:31:00.705 INFO:teuthology.orchestra.run.smithi146.stdout:Modify: 2024-09-15 20:23:10.522781853 +0000 2024-09-15T20:31:00.705 INFO:teuthology.orchestra.run.smithi146.stdout:Change: 2024-09-15 20:23:10.522781853 +0000 2024-09-15T20:31:00.705 INFO:teuthology.orchestra.run.smithi146.stdout: Birth: - 2024-09-15T20:31:00.705 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-15T20:31:00.759 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records in 2024-09-15T20:31:00.759 INFO:teuthology.orchestra.run.smithi146.stderr:1+0 records out 2024-09-15T20:31:00.759 INFO:teuthology.orchestra.run.smithi146.stderr:512 bytes copied, 0.000400667 s, 1.3 MB/s 2024-09-15T20:31:00.760 DEBUG:teuthology.orchestra.run.smithi146:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-15T20:31:00.810 INFO:tasks.ceph:osd dev map: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T20:31:00.810 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T20:31:00.810 INFO:tasks.ceph:Generating config... 2024-09-15T20:31:00.811 INFO:tasks.ceph:[client] debug monc = 20 2024-09-15T20:31:00.811 INFO:tasks.ceph:[client] debug ms = 1 2024-09-15T20:31:00.812 INFO:tasks.ceph:[global] debug ms = 1 2024-09-15T20:31:00.812 INFO:tasks.ceph:[global] debug objecter = 20 2024-09-15T20:31:00.812 INFO:tasks.ceph:[global] debug rados = 20 2024-09-15T20:31:00.812 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-15T20:31:00.812 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-09-15T20:31:00.812 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-09-15T20:31:00.812 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-15T20:31:00.812 INFO:tasks.ceph:[global] ms type = async 2024-09-15T20:31:00.812 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-15T20:31:00.812 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-15T20:31:00.812 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-15T20:31:00.812 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-15T20:31:00.812 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-15T20:31:00.812 INFO:tasks.ceph:[mon] mon min osdmap epochs = 25 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] mon scrub inject crc mismatch = 0.01 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] mon scrub inject missing keys = 0.05 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] paxos min = 10 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] paxos service trim min = 5 2024-09-15T20:31:00.813 INFO:tasks.ceph:[mon] paxos trim min = 10 2024-09-15T20:31:00.813 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-15T20:31:00.813 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-09-15T20:31:00.813 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-09-15T20:31:00.813 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] bluestore write v2 = True 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] osd class default list = * 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] osd class load list = * 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-09-15T20:31:00.814 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-15T20:31:00.815 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-09-15T20:31:00.815 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-09-15T20:31:00.815 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-15T20:31:00.815 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-09-15T20:31:00.815 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-09-15T20:31:00.815 INFO:tasks.ceph:Setting up mon.a... 2024-09-15T20:31:00.815 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-09-15T20:31:00.850 INFO:teuthology.orchestra.run.smithi078.stdout:creating /etc/ceph/ceph.keyring 2024-09-15T20:31:00.863 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-09-15T20:31:00.935 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T20:31:00.987 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.78:6789]'), ('mon.c', '[v1:172.21.15.78:6790]'), ('mon.b', '[v1:172.21.15.146:6789]')] 2024-09-15T20:31:00.988 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.78:6789],[v1:172.21.15.78:6790],[v1:172.21.15.146:6789]', 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore write v2': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 25, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub inject crc mismatch': 0.01, 'mon scrub inject missing keys': 0.05, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos min': 10, 'paxos service trim min': 5, 'paxos trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-09-15T20:31:00.989 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:00.989 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T20:31:01.034 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.78:6789]' --addv c '[v1:172.21.15.78:6790]' --addv b '[v1:172.21.15.146:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:01.119 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:01.119 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: generated fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:01.119 INFO:teuthology.orchestra.run.smithi078.stdout:setting min_mon_release = quincy 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:epoch 0 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:min_mon_release 17 (quincy) 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:election_strategy: 1 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:01.120 INFO:teuthology.orchestra.run.smithi078.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-09-15T20:31:01.121 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-15T20:31:01.169 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID a8a3eb37-3a92-4568-8aad-3536499064e4... 2024-09-15T20:31:01.170 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T20:31:01.215 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout:[global] 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: chdir = "" 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: auth supported = cephx 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: filestore xattr use omap = true 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: mon clock drift allowed = 1.000 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: osd crush chooseleaf type = 0 2024-09-15T20:31:01.243 INFO:teuthology.orchestra.run.smithi078.stdout: auth debug = true 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on old message = true 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: ms die on bug = true 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: mon pg warn max object skew = 0 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default size = 2 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.244 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow primary affinity = true 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd allow pg remap = true 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on legacy crush tunables = false 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on crush straw calc version zero = false 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on no sortbitwise = false 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on osd down out interval zero = false 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on too few osds = false 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: mon_allow_pool_size_one = true 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: osd default data pool replay window = 5 2024-09-15T20:31:01.245 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: mon allow pool delete = true 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: mon cluster log file level = debug 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: debug asserts on shutdown = true 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: mon health detail to clog = false 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: mon host = "[v1:172.21.15.78:6789],[v1:172.21.15.78:6790],[v1:172.21.15.146:6789]" 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: debug objecter = 20 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: debug rados = 20 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: mon client directed command retry = 5 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: mon election default strategy = 1 2024-09-15T20:31:01.246 INFO:teuthology.orchestra.run.smithi078.stdout: ms bind msgr2 = False 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: ms inject socket failures = 5000 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: ms type = async 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: fsid = a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout:[osd] 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: osd journal size = 100 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub load threshold = 5.0 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: osd scrub max interval = 600 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock profile = high_recovery_ops 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: osd recover clone overlap = true 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: osd recovery max chunk = 1048576 2024-09-15T20:31:01.247 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug shutdown = true 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug op order = true 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify stray on activate = true 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug trim objects = true 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd open classes on start = true 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug pg log writeout = true 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd deep scrub update digest min age = 30 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: osd map max advance = 10 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: journal zero on create = true 2024-09-15T20:31:01.248 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: filestore ondisk finisher threads = 3 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: filestore apply finisher threads = 3 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: bdev debug aio = true 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug misdirected ops = true 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore block size = 96636764160 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore compression algorithm = snappy 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore compression mode = aggressive 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore fsck on mount = True 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore write v2 = True 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: bluestore zero block detection = True 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluefs = 20 2024-09-15T20:31:01.249 INFO:teuthology.orchestra.run.smithi078.stdout: debug bluestore = 20 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: debug osd = 20 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: debug rocksdb = 10 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd full ratio = 0.9 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd nearfull ratio = 0.8 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: osd class default list = * 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: osd class load list = * 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify cached snaps = True 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: osd debug verify missing on start = True 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: osd failsafe full ratio = 0.95 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: osd mclock override recovery settings = True 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: osd objectstore = bluestore 2024-09-15T20:31:01.250 INFO:teuthology.orchestra.run.smithi078.stdout: osd op queue = debug_random 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: osd op queue cut off = debug_random 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr] 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: debug mgr = 20 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: mgr/telemetry/nag = false 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout:[mon] 2024-09-15T20:31:01.251 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: debug mon = 20 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: debug paxos = 20 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: debug auth = 20 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: mon data avail warn = 5 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: mon mgr mkfs grace = 240 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min pgs per osd = 4 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd reporter subtree level = osd 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: mon osd prime pg temp = true 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: mon reweight min bytes per osd = 10 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T20:31:01.252 INFO:teuthology.orchestra.run.smithi078.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: # 1m isn't quite enough 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon_down_mkfs_grace = 2m 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon_warn_on_filestore_osds = false 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon min osdmap epochs = 25 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon osdmap full prune interval = 2 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon osdmap full prune min = 15 2024-09-15T20:31:01.253 INFO:teuthology.orchestra.run.smithi078.stdout: mon osdmap full prune txsize = 2 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: mon scrub inject missing keys = 0.05 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: mon scrub interval = 300 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: mon warn on pool no app = False 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: paxos min = 10 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: paxos service trim min = 5 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: paxos trim min = 10 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout:[client] 2024-09-15T20:31:01.254 INFO:teuthology.orchestra.run.smithi078.stdout: rgw cache enabled = true 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable ops log = true 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout: rgw enable usage log = true 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout: debug monc = 20 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout: debug ms = 1 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.a] 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.c] 2024-09-15T20:31:01.256 INFO:teuthology.orchestra.run.smithi078.stdout:[mon.b] 2024-09-15T20:31:01.258 INFO:teuthology.orchestra.run.smithi146.stdout:[global] 2024-09-15T20:31:01.258 INFO:teuthology.orchestra.run.smithi146.stdout: chdir = "" 2024-09-15T20:31:01.258 INFO:teuthology.orchestra.run.smithi146.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-15T20:31:01.258 INFO:teuthology.orchestra.run.smithi146.stdout: auth supported = cephx 2024-09-15T20:31:01.258 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: filestore xattr use omap = true 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: mon clock drift allowed = 1.000 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: osd crush chooseleaf type = 0 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: auth debug = true 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on old message = true 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: ms die on bug = true 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: mon pg warn max object skew = 0 2024-09-15T20:31:01.259 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: # disable pg_autoscaler by default for new pools 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default size = 2 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow primary affinity = true 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd allow pg remap = true 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on legacy crush tunables = false 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on crush straw calc version zero = false 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on no sortbitwise = false 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on osd down out interval zero = false 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on too few osds = false 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-15T20:31:01.260 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: mon_allow_pool_size_one = true 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: osd default data pool replay window = 5 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: mon allow pool delete = true 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: mon cluster log file level = debug 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: debug asserts on shutdown = true 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: mon health detail to clog = false 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: mon host = "[v1:172.21.15.78:6789],[v1:172.21.15.78:6790],[v1:172.21.15.146:6789]" 2024-09-15T20:31:01.261 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: debug objecter = 20 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: debug rados = 20 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: mon client directed command retry = 5 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: mon election default strategy = 1 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: ms bind msgr2 = False 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: ms inject socket failures = 5000 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: ms type = async 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: fsid = a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout:[osd] 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: osd journal size = 100 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.262 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub load threshold = 5.0 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd scrub max interval = 600 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock profile = high_recovery_ops 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd recover clone overlap = true 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd recovery max chunk = 1048576 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug shutdown = true 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug op order = true 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify stray on activate = true 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug trim objects = true 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd open classes on start = true 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug pg log writeout = true 2024-09-15T20:31:01.263 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: osd deep scrub update digest min age = 30 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: osd map max advance = 10 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: journal zero on create = true 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: filestore ondisk finisher threads = 3 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: filestore apply finisher threads = 3 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: bdev debug aio = true 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug misdirected ops = true 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore block size = 96636764160 2024-09-15T20:31:01.264 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore compression algorithm = snappy 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore compression mode = aggressive 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore fsck on mount = True 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore write v2 = True 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: bluestore zero block detection = True 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluefs = 20 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: debug bluestore = 20 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: debug osd = 20 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: debug rocksdb = 10 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd backfillfull_ratio = 0.85 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd full ratio = 0.9 2024-09-15T20:31:01.265 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd nearfull ratio = 0.8 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd class default list = * 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd class load list = * 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify cached snaps = True 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd debug verify missing on start = True 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd failsafe full ratio = 0.95 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd mclock override recovery settings = True 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd objectstore = bluestore 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd op queue = debug_random 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: osd op queue cut off = debug_random 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr] 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-15T20:31:01.266 INFO:teuthology.orchestra.run.smithi146.stdout: debug mgr = 20 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: mgr/telemetry/nag = false 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout:[mon] 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: debug mon = 20 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: debug paxos = 20 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: debug auth = 20 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: mon data avail warn = 5 2024-09-15T20:31:01.267 INFO:teuthology.orchestra.run.smithi146.stdout: mon mgr mkfs grace = 240 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min pgs per osd = 4 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd reporter subtree level = osd 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: mon osd prime pg temp = true 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: mon reweight min bytes per osd = 10 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: auth mon ticket ttl = 660 # 11m 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: auth service ticket ttl = 240 # 4m 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: # don't complain about insecure global_id in the test suite 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-15T20:31:01.268 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: # 1m isn't quite enough 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon_down_mkfs_grace = 2m 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon_warn_on_filestore_osds = false 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon min osdmap epochs = 25 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune interval = 2 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune min = 15 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon osdmap full prune txsize = 2 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon scrub inject crc mismatch = 0.01 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon scrub inject missing keys = 0.05 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon scrub interval = 300 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: mon warn on pool no app = False 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: paxos min = 10 2024-09-15T20:31:01.269 INFO:teuthology.orchestra.run.smithi146.stdout: paxos service trim min = 5 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: paxos trim min = 10 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout:[client] 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: rgw cache enabled = true 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable ops log = true 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: rgw enable usage log = true 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: debug monc = 20 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout: debug ms = 1 2024-09-15T20:31:01.270 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.a] 2024-09-15T20:31:01.271 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.c] 2024-09-15T20:31:01.271 INFO:teuthology.orchestra.run.smithi146.stdout:[mon.b] 2024-09-15T20:31:01.271 INFO:tasks.ceph:Creating admin key on mon.a... 2024-09-15T20:31:01.271 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-09-15T20:31:01.350 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-15T20:31:01.350 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:01.350 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-09-15T20:31:01.355 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:01.356 DEBUG:teuthology.orchestra.run.smithi078:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-09-15T20:31:01.405 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:31:01.405 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:01.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T20:31:01.405 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T20:31:01.470 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:01.470 DEBUG:teuthology.orchestra.run.smithi078:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:01.517 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:31:01.518 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:01.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd of=/etc/ceph/ceph.keyring 2024-09-15T20:31:01.518 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-09-15T20:31:01.539 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:01.540 DEBUG:teuthology.orchestra.run.smithi146:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:01.589 INFO:tasks.ceph:Setting up mon nodes... 2024-09-15T20:31:01.589 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-15T20:31:01.589 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T20:31:01.674 INFO:teuthology.orchestra.run.smithi146.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-09-15T20:31:01.685 INFO:tasks.ceph:Setting up mds nodes... 2024-09-15T20:31:01.685 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-15T20:31:01.685 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-15T20:31:01.760 INFO:teuthology.orchestra.run.smithi146.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-09-15T20:31:01.771 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-15T20:31:01.771 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi078.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi146.front.sepia.ceph.com'): {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'}} 2024-09-15T20:31:01.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-09-15T20:31:01.785 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T20:31:01.785 INFO:tasks.ceph:role: osd.0 2024-09-15T20:31:01.785 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:31:01.785 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:31:01.850 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:31:02.335 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-09-15T20:31:02.337 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-09-15T20:31:02.337 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-09-15T20:31:02.492 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-09-15T20:31:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T20:31:02.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:31:02.504 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-09-15T20:31:02.557 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T20:31:02.557 INFO:tasks.ceph:role: osd.1 2024-09-15T20:31:02.557 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:31:02.557 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T20:31:02.620 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:31:02.620 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:31:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:31:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-15T20:31:02.621 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:31:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:31:02.621 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:31:02.621 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T20:31:02.621 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:31:02.621 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:31:03.056 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-09-15T20:31:03.058 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-09-15T20:31:03.058 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-09-15T20:31:03.143 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-09-15T20:31:03.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:31:03.195 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T20:31:03.196 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-09-15T20:31:03.248 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2024-09-15T20:31:03.249 INFO:tasks.ceph:role: osd.2 2024-09-15T20:31:03.249 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:31:03.249 DEBUG:teuthology.orchestra.run.smithi078:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T20:31:03.313 INFO:teuthology.orchestra.run.smithi078.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:31:03.313 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:31:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:31:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-15T20:31:03.314 INFO:teuthology.orchestra.run.smithi078.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:31:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:31:03.314 INFO:teuthology.orchestra.run.smithi078.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:31:03.314 INFO:teuthology.orchestra.run.smithi078.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T20:31:03.314 INFO:teuthology.orchestra.run.smithi078.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:31:03.314 INFO:teuthology.orchestra.run.smithi078.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:31:03.769 INFO:teuthology.orchestra.run.smithi078.stdout:Discarding blocks...Done. 2024-09-15T20:31:03.771 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi078.front.sepia.ceph.com -o noatime 2024-09-15T20:31:03.771 DEBUG:teuthology.orchestra.run.smithi078:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-09-15T20:31:03.882 DEBUG:teuthology.orchestra.run.smithi078:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-09-15T20:31:03.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:31:03.893 INFO:teuthology.orchestra.run.smithi078.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T20:31:03.894 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:04.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:03.997+0000 7f28d048e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-09-15T20:31:04.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:04.005+0000 7f28d048e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-09-15T20:31:04.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:04.005+0000 7f28d048e7c0 -1 bdev(0x55b2d6b71800 /var/lib/ceph/osd/ceph-0/block) open stat got: (1) Operation not permitted 2024-09-15T20:31:04.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:04.005+0000 7f28d048e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-09-15T20:31:07.154 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-15T20:31:07.168 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:07.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:07.245+0000 7f508699b7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-09-15T20:31:07.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:07.245+0000 7f508699b7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-09-15T20:31:07.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:07.245+0000 7f508699b7c0 -1 bdev(0x55ad14c69800 /var/lib/ceph/osd/ceph-1/block) open stat got: (1) Operation not permitted 2024-09-15T20:31:07.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:07.245+0000 7f508699b7c0 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-09-15T20:31:10.394 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-15T20:31:10.452 DEBUG:teuthology.orchestra.run.smithi078:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:10.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:10.528+0000 7fd2f8d3c7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-09-15T20:31:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:10.528+0000 7fd2f8d3c7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-09-15T20:31:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:10.528+0000 7fd2f8d3c7c0 -1 bdev(0x56242a645800 /var/lib/ceph/osd/ceph-2/block) open stat got: (1) Operation not permitted 2024-09-15T20:31:10.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:10.528+0000 7fd2f8d3c7c0 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-09-15T20:31:13.667 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-15T20:31:13.722 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-09-15T20:31:13.737 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T20:31:13.737 INFO:tasks.ceph:role: osd.3 2024-09-15T20:31:13.737 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:31:13.737 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-15T20:31:13.800 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:31:13.801 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:31:14.292 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-09-15T20:31:14.294 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-09-15T20:31:14.294 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-3 2024-09-15T20:31:14.458 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-09-15T20:31:14.469 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T20:31:14.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:31:14.470 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-09-15T20:31:14.524 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T20:31:14.524 INFO:tasks.ceph:role: osd.4 2024-09-15T20:31:14.524 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:31:14.524 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-15T20:31:14.589 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:31:14.589 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:31:14.589 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:31:14.589 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-15T20:31:14.589 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:31:14.589 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:31:14.589 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:31:14.590 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T20:31:14.590 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:31:14.590 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:31:15.022 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-09-15T20:31:15.024 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-09-15T20:31:15.024 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-4 2024-09-15T20:31:15.117 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-09-15T20:31:15.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:31:15.127 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T20:31:15.127 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-09-15T20:31:15.180 INFO:tasks.ceph:roles_to_devs: {'osd.3': '/dev/vg_nvme/lv_1', 'osd.4': '/dev/vg_nvme/lv_2', 'osd.5': '/dev/vg_nvme/lv_3'} 2024-09-15T20:31:15.180 INFO:tasks.ceph:role: osd.5 2024-09-15T20:31:15.180 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:31:15.180 DEBUG:teuthology.orchestra.run.smithi146:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout: = sunit=0 swidth=0 blks 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-15T20:31:15.240 INFO:teuthology.orchestra.run.smithi146.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-15T20:31:15.708 INFO:teuthology.orchestra.run.smithi146.stdout:Discarding blocks...Done. 2024-09-15T20:31:15.710 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi146.front.sepia.ceph.com -o noatime 2024-09-15T20:31:15.710 DEBUG:teuthology.orchestra.run.smithi146:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5 2024-09-15T20:31:15.842 DEBUG:teuthology.orchestra.run.smithi146:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-09-15T20:31:15.852 INFO:teuthology.orchestra.run.smithi146.stderr:sudo: /sbin/restorecon: command not found 2024-09-15T20:31:15.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:31:15.853 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:16.078 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:16.074+0000 7f262f89e7c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-09-15T20:31:16.082 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:16.078+0000 7f262f89e7c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-09-15T20:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:16.078+0000 7f262f89e7c0 -1 bdev(0x563a5e18f800 /var/lib/ceph/osd/ceph-3/block) open stat got: (1) Operation not permitted 2024-09-15T20:31:16.083 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:16.078+0000 7f262f89e7c0 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-09-15T20:31:19.227 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-15T20:31:19.241 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:19.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:19.318+0000 7fcf0a8577c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-09-15T20:31:19.323 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:19.318+0000 7fcf0a8577c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-09-15T20:31:19.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:19.318+0000 7fcf0a8577c0 -1 bdev(0x556bb0a5b800 /var/lib/ceph/osd/ceph-4/block) open stat got: (1) Operation not permitted 2024-09-15T20:31:19.324 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:19.318+0000 7fcf0a8577c0 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-09-15T20:31:22.448 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-15T20:31:22.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:22.587 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:22.582+0000 7fb4925367c0 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-09-15T20:31:22.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:22.582+0000 7fb4925367c0 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-09-15T20:31:22.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:22.582+0000 7fb4925367c0 -1 bdev(0x55e017b49800 /var/lib/ceph/osd/ceph-5/block) open stat got: (1) Operation not permitted 2024-09-15T20:31:22.588 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:22.582+0000 7fb4925367c0 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-09-15T20:31:25.763 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-15T20:31:25.818 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-15T20:31:25.818 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:25.818 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-09-15T20:31:25.831 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:25.831 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-09-15T20:31:25.885 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:25.885 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-09-15T20:31:25.940 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:25.940 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-09-15T20:31:25.953 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:25.953 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-09-15T20:31:26.008 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:26.009 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-09-15T20:31:26.064 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:26.064 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-09-15T20:31:26.116 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:26.116 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-09-15T20:31:26.161 INFO:tasks.ceph:Adding keys to all mons... 2024-09-15T20:31:26.162 DEBUG:teuthology.orchestra.run.smithi078:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T20:31:26.164 DEBUG:teuthology.orchestra.run.smithi146:> sudo tee -a /etc/ceph/ceph.keyring 2024-09-15T20:31:26.207 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.0] 2024-09-15T20:31:26.207 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAIROdmB/xHABAAE/N2n9P4CO8IRKELksUUqw== 2024-09-15T20:31:26.207 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.1] 2024-09-15T20:31:26.207 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQALROdmKY77DhAAJ1jdPZitEcM3QSTG5S6hPg== 2024-09-15T20:31:26.207 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.2] 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAOROdm2AvrHxAA/0GEW8yz0mcRuLsWudE0bg== 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout:[mgr.x] 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAFROdmFEwqKBAACNS2Ec3RG4yxclUNS6vCIA== 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.3] 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAUROdm8j2YBBAAWAJn6jAdv0fNQ8rq9+Vptg== 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.4] 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAXROdmDTw3ExAAA5Rf99DW89IGRxi9wI7yBg== 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout:[osd.5] 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAaROdmZez2IhAAD6mSpfC1v8wz982Ea4CkaA== 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout:[client.0] 2024-09-15T20:31:26.208 INFO:teuthology.orchestra.run.smithi078.stdout: key = AQAFROdmR7pJLRAAX6mAA8SNt7ICcAGHtZcGOA== 2024-09-15T20:31:26.215 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.0] 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAIROdmB/xHABAAE/N2n9P4CO8IRKELksUUqw== 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.1] 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQALROdmKY77DhAAJ1jdPZitEcM3QSTG5S6hPg== 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.2] 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAOROdm2AvrHxAA/0GEW8yz0mcRuLsWudE0bg== 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout:[mgr.x] 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAFROdmFEwqKBAACNS2Ec3RG4yxclUNS6vCIA== 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.3] 2024-09-15T20:31:26.216 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAUROdm8j2YBBAAWAJn6jAdv0fNQ8rq9+Vptg== 2024-09-15T20:31:26.217 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.4] 2024-09-15T20:31:26.217 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAXROdmDTw3ExAAA5Rf99DW89IGRxi9wI7yBg== 2024-09-15T20:31:26.217 INFO:teuthology.orchestra.run.smithi146.stdout:[osd.5] 2024-09-15T20:31:26.217 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAaROdmZez2IhAAD6mSpfC1v8wz982Ea4CkaA== 2024-09-15T20:31:26.217 INFO:teuthology.orchestra.run.smithi146.stdout:[client.0] 2024-09-15T20:31:26.217 INFO:teuthology.orchestra.run.smithi146.stdout: key = AQAFROdmR7pJLRAAX6mAA8SNt7ICcAGHtZcGOA== 2024-09-15T20:31:26.218 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.255 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.309 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.343 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.387 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.389 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.466 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T20:31:26.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-15T20:31:26.551 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.553 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.637 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.639 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.721 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.731 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-15T20:31:26.806 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T20:31:26.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-15T20:31:26.892 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-15T20:31:26.892 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-09-15T20:31:26.906 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T20:31:26.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-09-15T20:31:27.008 DEBUG:teuthology.orchestra.run.smithi078:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-09-15T20:31:27.058 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T20:31:27.149 DEBUG:teuthology.orchestra.run.smithi078:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-09-15T20:31:27.203 DEBUG:teuthology.orchestra.run.smithi146:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-09-15T20:31:27.217 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-09-15T20:31:27.310 DEBUG:teuthology.orchestra.run.smithi146:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-09-15T20:31:27.321 DEBUG:teuthology.orchestra.run.smithi078:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:27.323 DEBUG:teuthology.orchestra.run.smithi146:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-09-15T20:31:27.373 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-09-15T20:31:27.373 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T20:31:27.373 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T20:31:27.376 INFO:tasks.ceph.mon.a:Started 2024-09-15T20:31:27.376 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T20:31:27.376 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T20:31:27.378 INFO:tasks.ceph.mon.c:Started 2024-09-15T20:31:27.378 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T20:31:27.378 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T20:31:27.418 INFO:tasks.ceph.mon.b:Started 2024-09-15T20:31:27.418 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-09-15T20:31:27.418 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-09-15T20:31:27.419 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-09-15T20:31:27.421 INFO:tasks.ceph.mgr.x:Started 2024-09-15T20:31:27.421 DEBUG:tasks.ceph:set 0 configs 2024-09-15T20:31:27.421 INFO:tasks.ceph:Setting crush tunables to default 2024-09-15T20:31:27.421 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd crush tunables default 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.532+0000 7f8df5f0b640 10 monclient: get_monmap_and_config 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.532+0000 7f8df5f0b640 10 monclient: build_initial_monmap 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.532+0000 7f8df5f0b640 10 monclient: monmap: 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:27.536541+0000 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:27.536541+0000 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:27.537 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:27.538 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:27.538 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 1 Processor -- start 2024-09-15T20:31:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 1 -- start start 2024-09-15T20:31:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 10 monclient: init 2024-09-15T20:31:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 10 monclient(hunting): picked mon.noname-b con 0x7f8df0137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 10 monclient(hunting): picked mon.noname-a con 0x7f8df0074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:27.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 10 monclient(hunting): picked mon.noname-c con 0x7f8df013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df0138850 con 0x7f8df0074500 2024-09-15T20:31:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df00a5da0 con 0x7f8df013e370 2024-09-15T20:31:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df0073560 con 0x7f8df0137d60 2024-09-15T20:31:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8df5f0b640 10 monclient(hunting): authenticate will time out at 1148.420869s 2024-09-15T20:31:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8def7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f8df0137d60 0x7f8df0138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43720/0 (socket says 172.21.15.78:43720) 2024-09-15T20:31:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8de7fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8df0074500 0x7f8df013cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39658/0 (socket says 172.21.15.78:39658) 2024-09-15T20:31:27.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.536+0000 7f8def7fe640 1 -- 172.21.15.78:0/3982151845 learned_addr learned my addr 172.21.15.78:0/3982151845 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2797945274 0 0) 0x7f8df0138850 con 0x7f8df0074500 2024-09-15T20:31:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): my global_id is 4098 2024-09-15T20:31:27.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ddc003530 con 0x7f8df0074500 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1990564040 0 0) 0x7f8df0073560 con 0x7f8df0137d60 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): my global_id is 4103 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df0138850 con 0x7f8df0137d60 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1068379765 0 0) 0x7f8df00a5da0 con 0x7f8df013e370 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient(hunting): my global_id is 4102 2024-09-15T20:31:27.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df0073560 con 0x7f8df013e370 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3365071858 0 0) 0x7f8df0138850 con 0x7f8df0137d60 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 >> v1:172.21.15.146:6789/0 conn(0x7f8df013e370 legacy=0x7f8df0140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 >> v1:172.21.15.78:6789/0 conn(0x7f8df0074500 legacy=0x7f8df013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 monclient: found mon.noname-b 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df0073840 con 0x7f8df0137d60 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: _finish_auth 0 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8df5f0b640 5 monclient: authenticate success, global_id 4103 2024-09-15T20:31:27.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8de0001680 con 0x7f8df0137d60 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8df5f0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: dump: 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:27.728 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8df5f0b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de0001820 con 0x7f8df0137d60 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 1 -- 172.21.15.78:0/3982151845 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8de0001bf0 con 0x7f8df0137d60 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8df5f0b640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8df5f0b640 4 set_mon_vals no callback set 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.720+0000 7f8deeffd640 10 monclient: dump: 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.729 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:27.730 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:27.730 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:27.730 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:27.730 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:27.730 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 10 monclient: shutdown 2024-09-15T20:31:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 1 -- 172.21.15.78:0/3982151845 >> v1:172.21.15.78:6790/0 conn(0x7f8df0137d60 legacy=0x7f8df0138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 1 -- 172.21.15.78:0/3982151845 shutdown_connections 2024-09-15T20:31:27.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 1 -- 172.21.15.78:0/3982151845 >> 172.21.15.78:0/3982151845 conn(0x7f8df00aa650 msgr2=0x7f8df00aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 1 -- 172.21.15.78:0/3982151845 shutdown_connections 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 1 -- 172.21.15.78:0/3982151845 wait complete. 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 10 monclient: build_initial_monmap 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.724+0000 7f8df5f0b640 10 monclient: monmap: 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:27.735 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 librados: starting msgr at 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 librados: starting objecter 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 Processor -- start 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 -- start start 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 librados: setting wanted keys 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 librados: calling monclient init 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient: init 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient(hunting): picked mon.noname-c con 0x7f8df0074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient(hunting): picked mon.noname-a con 0x7f8df0137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient(hunting): picked mon.noname-b con 0x7f8df013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:27.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df0139820 con 0x7f8df0137d60 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df013a3b0 con 0x7f8df013e370 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8df0141740 con 0x7f8df0074500 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient(hunting): authenticate will time out at 1148.613609s 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8def7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f8df0074500 0x7f8df0138a00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43744/0 (socket says 172.21.15.78:43744) 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8def7fe640 1 -- 172.21.15.78:0/2334203754 learned_addr learned my addr 172.21.15.78:0/2334203754 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 client.?.objecter ms_handle_connect 0x7f8df0074500 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 client.?.objecter ms_handle_connect 0x7f8df0137d60 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 client.?.objecter ms_handle_connect 0x7f8df013e370 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1777538881 0 0) 0x7f8df013a3b0 con 0x7f8df013e370 2024-09-15T20:31:27.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): my global_id is 4105 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8dd0004ec0 con 0x7f8df013e370 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1500393404 0 0) 0x7f8df0141740 con 0x7f8df0074500 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): my global_id is 4106 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df013a3b0 con 0x7f8df0074500 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1436769315 0 0) 0x7f8dd0004ec0 con 0x7f8df013e370 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8df0141740 con 0x7f8df013e370 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 80175959 0 0) 0x7f8df013a3b0 con 0x7f8df0074500 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8dd0004ec0 con 0x7f8df0074500 2024-09-15T20:31:27.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1581144336 0 0) 0x7f8df0139820 con 0x7f8df0137d60 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): my global_id is 4107 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8df013a3b0 con 0x7f8df0137d60 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8de8001c80 con 0x7f8df013e370 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): dump: 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:27.739 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8de0001b70 con 0x7f8df0074500 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient(hunting): dump: 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.740 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2789253541 0 0) 0x7f8df0141740 con 0x7f8df013e370 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 >> v1:172.21.15.78:6790/0 conn(0x7f8df0074500 legacy=0x7f8df0138a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 >> v1:172.21.15.78:6789/0 conn(0x7f8df0137d60 legacy=0x7f8df0139110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient: _finish_hunting 0 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 monclient: found mon.b 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:27.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8df013be70 con 0x7f8df013e370 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient: _finish_auth 0 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient: _check_auth_tickets 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 5 monclient: authenticate success, global_id 4105 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient: _renew_subs 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8df013c150 con 0x7f8df013e370 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 client.4105.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient: _renew_subs 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8df013e0a0 con 0x7f8df013e370 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8df5f0b640 1 librados: init done 2024-09-15T20:31:27.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8de80039a0 con 0x7f8df013e370 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8de8003cf0 con 0x7f8df013e370 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8dee7fc640 4 set_mon_vals no callback set 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 monclient: dump: 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:27.743 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8de8004220 con 0x7f8df013e370 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (unknown 3532720460 0 0) 0x7f8de8005850 con 0x7f8df013e370 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 10 client.4105.objecter ms_dispatch 0x7f8df0000ba0 osd_map(1..1 src has 1..1) 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 3 client.4105.objecter handle_osd_map got epochs [1,1] > 0 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.728+0000 7f8decff9640 3 client.4105.objecter handle_osd_map decoding full epoch 1 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.732+0000 7f8decff9640 20 client.4105.objecter dump_active .. 0 homeless 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.732+0000 7f8df5f0b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.732+0000 7f8df5f0b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.732+0000 7f8df5f0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:27.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.732+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dd4005090 con 0x7f8df013e370 2024-09-15T20:31:27.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.732+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f8de8004d40 con 0x7f8df013e370 2024-09-15T20:31:27.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.732+0000 7f8decff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:27.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.732+0000 7f8decff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:27.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.800+0000 7f8df5f0b640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T20:31:27.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.800+0000 7f8df5f0b640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T20:31:27.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.800+0000 7f8df5f0b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:27.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.800+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f8dd4005380 con 0x7f8df013e370 2024-09-15T20:31:27.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8decff9640 1 -- 172.21.15.78:0/2334203754 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (unknown 2277712955 0 0) 0x7f8de8004650 con 0x7f8df013e370 2024-09-15T20:31:27.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8decff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-09-15T20:31:27.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8decff9640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-09-15T20:31:27.810 INFO:teuthology.orchestra.run.smithi078.stderr:adjusted tunables profile to default 2024-09-15T20:31:27.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 10 librados: watch_flush enter 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 10 librados: watch_flush exit 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 20 client.4105.objecter shutdown clearing up homeless session... 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 10 client.4105.objecter successfully canceled tick 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 10 monclient: shutdown 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 >> v1:172.21.15.146:6789/0 conn(0x7f8df013e370 legacy=0x7f8df01d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 shutdown_connections 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 >> 172.21.15.78:0/2334203754 conn(0x7f8df00aa650 msgr2=0x7f8df00a28b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 shutdown_connections 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 1 -- 172.21.15.78:0/2334203754 wait complete. 2024-09-15T20:31:27.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:27.808+0000 7f8df5f0b640 1 librados: shutdown 2024-09-15T20:31:27.828 INFO:tasks.ceph:check_enable_crimson: False 2024-09-15T20:31:27.829 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-09-15T20:31:27.829 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:27.829 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-09-15T20:31:27.841 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:27.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-09-15T20:31:27.896 DEBUG:teuthology.orchestra.run.smithi078:> set -ex 2024-09-15T20:31:27.896 DEBUG:teuthology.orchestra.run.smithi078:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-09-15T20:31:27.948 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:27.948 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-09-15T20:31:27.962 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:27.962 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-09-15T20:31:28.020 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:31:28.020 DEBUG:teuthology.orchestra.run.smithi146:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-09-15T20:31:28.076 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 9003b16b-6a87-4659-8436-ff541ea8dfcb 0 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient: get_monmap_and_config 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient: build_initial_monmap 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient: monmap: 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:28.241389+0000 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:28.241389+0000 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:28.243 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:28.244 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 1 Processor -- start 2024-09-15T20:31:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 1 -- start start 2024-09-15T20:31:28.244 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient: init 2024-09-15T20:31:28.251 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient(hunting): picked mon.noname-c con 0x7f8658137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient(hunting): picked mon.noname-a con 0x7f86580a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient(hunting): picked mon.noname-b con 0x7f865813e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580a6420 con 0x7f86580a9610 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580a6700 con 0x7f8658137ce0 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580a2110 con 0x7f865813e5f0 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865f7ba640 10 monclient(hunting): authenticate will time out at 1138.151206s 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865d556640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f8658137ce0 0x7f86581380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49820/0 (socket says 172.21.15.146:49820) 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f865d556640 1 -- 172.21.15.146:0/912699737 learned_addr learned my addr 172.21.15.146:0/912699737 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:28.252 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2696016033 0 0) 0x7f86580a6700 con 0x7f8658137ce0 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 10 monclient(hunting): my global_id is 4108 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8644003530 con 0x7f8658137ce0 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2901127534 0 0) 0x7f8644003530 con 0x7f8658137ce0 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 >> v1:172.21.15.78:6790/0 conn(0x7f865813e5f0 legacy=0x7f86581409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 >> v1:172.21.15.78:6789/0 conn(0x7f86580a9610 legacy=0x7f865813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 1 monclient: found mon.noname-c 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86580a4bc0 con 0x7f8658137ce0 2024-09-15T20:31:28.253 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 10 monclient: _finish_auth 0 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 5 monclient: authenticate success, global_id 4108 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.238+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86540016a0 con 0x7f8658137ce0 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 10 monclient: dump: 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.254 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8654001900 con 0x7f8658137ce0 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 1 -- 172.21.15.146:0/912699737 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8654001d10 con 0x7f8658137ce0 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 4 set_mon_vals no callback set 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864ffff640 10 monclient: dump: 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.255 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient: shutdown 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- 172.21.15.146:0/912699737 >> v1:172.21.15.146:6789/0 conn(0x7f8658137ce0 legacy=0x7f86581380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- 172.21.15.146:0/912699737 shutdown_connections 2024-09-15T20:31:28.256 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- 172.21.15.146:0/912699737 >> 172.21.15.146:0/912699737 conn(0x7f86580aa6c0 msgr2=0x7f86580aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- 172.21.15.146:0/912699737 shutdown_connections 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- 172.21.15.146:0/912699737 wait complete. 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient: build_initial_monmap 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient: monmap: 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.257 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 librados: starting msgr at 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 librados: starting objecter 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 Processor -- start 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- start start 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 librados: setting wanted keys 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 librados: calling monclient init 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient: init 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient(hunting): picked mon.noname-a con 0x7f86580a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient(hunting): picked mon.noname-c con 0x7f8658137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient(hunting): picked mon.noname-b con 0x7f865813e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:28.258 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865813ab90 con 0x7f86580a9610 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f865813c0c0 con 0x7f865813e5f0 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86580a9270 con 0x7f8658137ce0 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865f7ba640 10 monclient(hunting): authenticate will time out at 1138.154207s 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865dd57640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f865813e5f0 0x7f86581d5000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49830/0 (socket says 172.21.15.146:49830) 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865dd57640 1 -- 172.21.15.146:0/2239308317 learned_addr learned my addr 172.21.15.146:0/2239308317 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 client.?.objecter ms_handle_connect 0x7f865813e5f0 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865dd57640 1 -- 172.21.15.146:0/2239308317 >> v1:172.21.15.146:6789/0 conn(0x7f865813e5f0 legacy=0x7f86581d5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865dd57640 1 -- 172.21.15.146:0/2239308317 >> v1:172.21.15.146:6789/0 conn(0x7f865813e5f0 legacy=0x7f86581d5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865dd57640 1 --1- 172.21.15.146:0/2239308317 >> v1:172.21.15.146:6789/0 conn(0x7f865813e5f0 0x7f86581d5000 :-1 s=OPENED pgs=15 cs=1 l=1).handle_message read tag failed 2024-09-15T20:31:28.259 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f865dd57640 1 --1- 172.21.15.146:0/2239308317 >> v1:172.21.15.146:6789/0 conn(0x7f865813e5f0 0x7f86581d5000 :-1 s=OPENED pgs=15 cs=1 l=1).fault on lossy channel, failing 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.146:6789/0 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 client.?.objecter ms_handle_connect 0x7f8658137ce0 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 client.?.objecter ms_handle_connect 0x7f86580a9610 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2586621565 0 0) 0x7f86580a9270 con 0x7f8658137ce0 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): my global_id is 4112 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f863c004ec0 con 0x7f8658137ce0 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2795216002 0 0) 0x7f865813ab90 con 0x7f86580a9610 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.260 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): my global_id is 4110 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86580a9270 con 0x7f86580a9610 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2605510560 0 0) 0x7f86580a9270 con 0x7f86580a9610 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f865813ab90 con 0x7f86580a9610 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86540014c0 con 0x7f86580a9610 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): dump: 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.261 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1872188006 0 0) 0x7f863c004ec0 con 0x7f8658137ce0 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86580a9270 con 0x7f8658137ce0 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8648001760 con 0x7f8658137ce0 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.242+0000 7f864dffb640 10 monclient(hunting): dump: 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.262 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2669497828 0 0) 0x7f865813ab90 con 0x7f86580a9610 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 >> v1:172.21.15.78:6790/0 conn(0x7f8658137ce0 legacy=0x7f86581d18d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 >> v1:172.21.15.146:6789/0 conn(0x7f865813e5f0 legacy=0x7f86581d5000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 monclient: found mon.a 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:28.263 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f865813bde0 con 0x7f86580a9610 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 5 monclient: authenticate success, global_id 4110 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 10 monclient: _renew_subs 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86581c9c00 con 0x7f86580a9610 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 10 monclient: _renew_subs 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f86580a0040 con 0x7f86580a9610 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 1 librados: init done 2024-09-15T20:31:28.264 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86540016e0 con 0x7f86580a9610 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864f7fe640 4 set_mon_vals no callback set 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8654004e20 con 0x7f86580a9610 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 monclient: dump: 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.265 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8654005350 con 0x7f86580a9610 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (unknown 3006615258 0 0) 0x7f8654006610 con 0x7f86580a9610 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 10 client.4110.objecter ms_dispatch 0x7f8658000ba0 osd_map(2..2 src has 1..2) 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 3 client.4110.objecter handle_osd_map got epochs [2,2] > 0 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 3 client.4110.objecter handle_osd_map decoding full epoch 2 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f864dffb640 20 client.4110.objecter dump_active .. 0 homeless 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.246+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8638005090 con 0x7f86580a9610 2024-09-15T20:31:28.266 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.250+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f86540068b0 con 0x7f86580a9610 2024-09-15T20:31:28.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.250+0000 7f864dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.267 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.250+0000 7f864dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:28.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.322+0000 7f865f7ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "9003b16b-6a87-4659-8436-ff541ea8dfcb", "id": 0}] 2024-09-15T20:31:28.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.322+0000 7f865f7ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "9003b16b-6a87-4659-8436-ff541ea8dfcb", "id": 0}] 2024-09-15T20:31:28.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.322+0000 7f865f7ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:28.327 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.322+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd new", "uuid": "9003b16b-6a87-4659-8436-ff541ea8dfcb", "id": 0} v 0) -- 0x7f8638005380 con 0x7f86580a9610 2024-09-15T20:31:28.329 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f864dffb640 1 -- 172.21.15.146:0/2239308317 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9003b16b-6a87-4659-8436-ff541ea8dfcb", "id": 0}]=0 v3) ==== 112+0+2 (unknown 2213683943 0 732707515) 0x7f8654006b10 con 0x7f86580a9610 2024-09-15T20:31:28.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f864dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "9003b16b-6a87-4659-8436-ff541ea8dfcb", "id": 0}] 2024-09-15T20:31:28.330 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f864dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:28.330 INFO:teuthology.orchestra.run.smithi146.stdout:0 2024-09-15T20:31:28.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 10 librados: watch_flush enter 2024-09-15T20:31:28.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 10 librados: watch_flush exit 2024-09-15T20:31:28.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 20 client.4110.objecter shutdown clearing up homeless session... 2024-09-15T20:31:28.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 10 client.4110.objecter successfully canceled tick 2024-09-15T20:31:28.331 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 10 monclient: shutdown 2024-09-15T20:31:28.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:28.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 >> v1:172.21.15.78:6789/0 conn(0x7f86580a9610 legacy=0x7f86581c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 shutdown_connections 2024-09-15T20:31:28.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 >> 172.21.15.146:0/2239308317 conn(0x7f86580aa6c0 msgr2=0x7f8658073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:28.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 shutdown_connections 2024-09-15T20:31:28.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 1 -- 172.21.15.146:0/2239308317 wait complete. 2024-09-15T20:31:28.332 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.326+0000 7f865f7ba640 1 librados: shutdown 2024-09-15T20:31:28.347 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new be602c68-218b-4573-8d19-eef568af9d4e 1 2024-09-15T20:31:28.467 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.462+0000 7f92f5555640 10 monclient: get_monmap_and_config 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.462+0000 7f92f5555640 10 monclient: build_initial_monmap 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.462+0000 7f92f5555640 10 monclient: monmap: 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:28.466549+0000 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:28.466549+0000 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:28.468 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.471 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.462+0000 7f92f5555640 1 Processor -- start 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.462+0000 7f92f5555640 1 -- start start 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.462+0000 7f92f5555640 10 monclient: init 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): picked mon.noname-b con 0x7f92f00a1a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): picked mon.noname-c con 0x7f92f00a2f80 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): picked mon.noname-a con 0x7f92f00a2500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f009e5b0 con 0x7f92f00a2500 2024-09-15T20:31:28.472 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f009dc90 con 0x7f92f00a2f80 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f009ef90 con 0x7f92f00a1a10 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): authenticate will time out at 1138.376589s 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e7fff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f92f00a2f80 0x7f92f0139d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49846/0 (socket says 172.21.15.146:49846) 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e7fff640 1 -- 172.21.15.146:0/683563841 learned_addr learned my addr 172.21.15.146:0/683563841 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3367028734 0 0) 0x7f92f009dc90 con 0x7f92f00a2f80 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient(hunting): my global_id is 4114 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92dc003530 con 0x7f92f00a2f80 2024-09-15T20:31:28.473 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3720251278 0 0) 0x7f92dc003530 con 0x7f92f00a2f80 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 >> v1:172.21.15.78:6790/0 conn(0x7f92f00a1a10 legacy=0x7f92f00a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 >> v1:172.21.15.78:6789/0 conn(0x7f92f00a2500 legacy=0x7f92f013d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 monclient: found mon.noname-c 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f0141bf0 con 0x7f92f00a2f80 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 5 monclient: authenticate success, global_id 4114 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f92d80016a0 con 0x7f92f00a2f80 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:28.474 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: dump: 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92d8001940 con 0x7f92f00a2f80 2024-09-15T20:31:28.475 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 1 -- 172.21.15.146:0/683563841 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f92d8001d50 con 0x7f92f00a2f80 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 4 set_mon_vals no callback set 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92ee7fc640 10 monclient: dump: 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.476 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: shutdown 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- 172.21.15.146:0/683563841 >> v1:172.21.15.146:6789/0 conn(0x7f92f00a2f80 legacy=0x7f92f0139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- 172.21.15.146:0/683563841 shutdown_connections 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- 172.21.15.146:0/683563841 >> 172.21.15.146:0/683563841 conn(0x7f92f00a3ce0 msgr2=0x7f92f00a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- 172.21.15.146:0/683563841 shutdown_connections 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- 172.21.15.146:0/683563841 wait complete. 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: build_initial_monmap 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: monmap: 2024-09-15T20:31:28.477 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 librados: starting msgr at 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 librados: starting objecter 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 Processor -- start 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- start start 2024-09-15T20:31:28.478 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 librados: setting wanted keys 2024-09-15T20:31:28.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 librados: calling monclient init 2024-09-15T20:31:28.479 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: init 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): picked mon.noname-b con 0x7f92f00a1a10 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): picked mon.noname-c con 0x7f92f00a2500 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): picked mon.noname-a con 0x7f92f00a2f80 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f0073da0 con 0x7f92f00a2f80 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f012f250 con 0x7f92f00a1a10 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f92f012f530 con 0x7f92f00a2500 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92f5555640 10 monclient(hunting): authenticate will time out at 1138.379646s 2024-09-15T20:31:28.482 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92eeffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f92f00a1a10 0x7f92f0072f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49856/0 (socket says 172.21.15.146:49856) 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92eeffd640 1 -- 172.21.15.146:0/1356466764 learned_addr learned my addr 172.21.15.146:0/1356466764 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e77fe640 10 client.?.objecter ms_handle_connect 0x7f92f00a1a10 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127536460 0 0) 0x7f92f012f250 con 0x7f92f00a1a10 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e77fe640 10 monclient(hunting): my global_id is 4117 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.466+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92d0004ec0 con 0x7f92f00a1a10 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 client.?.objecter ms_handle_connect 0x7f92f00a2500 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 client.?.objecter ms_handle_connect 0x7f92f00a2f80 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2285799983 0 0) 0x7f92d0004ec0 con 0x7f92f00a1a10 2024-09-15T20:31:28.483 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92f012f250 con 0x7f92f00a1a10 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f92e0001970 con 0x7f92f00a1a10 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient(hunting): dump: 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.484 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1751733781 0 0) 0x7f92f012f250 con 0x7f92f00a1a10 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 >> v1:172.21.15.78:6790/0 conn(0x7f92f00a2500 legacy=0x7f92f0073690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 >> v1:172.21.15.78:6789/0 conn(0x7f92f00a2f80 legacy=0x7f92f01d2b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 monclient: found mon.b 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f012f810 con 0x7f92f00a1a10 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:28.485 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 5 monclient: authenticate success, global_id 4117 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 10 monclient: _renew_subs 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f92f012dc70 con 0x7f92f00a1a10 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 10 client.4117.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 10 monclient: _renew_subs 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f92f0074a90 con 0x7f92f00a1a10 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 1 librados: init done 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f92e0001cd0 con 0x7f92f00a1a10 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f92e0003a50 con 0x7f92f00a1a10 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92edffb640 4 set_mon_vals no callback set 2024-09-15T20:31:28.486 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: dump: 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f92e0004040 con 0x7f92f00a1a10 2024-09-15T20:31:28.487 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (unknown 3118383510 0 0) 0x7f92e0006770 con 0x7f92f00a1a10 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 client.4117.objecter ms_dispatch 0x7f92f0000ba0 osd_map(3..3 src has 1..3) 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 3 client.4117.objecter handle_osd_map got epochs [3,3] > 0 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 3 client.4117.objecter handle_osd_map decoding full epoch 3 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 20 client.4117.objecter dump_active .. 0 homeless 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92d4005090 con 0x7f92f00a1a10 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f92f0074a90 con 0x7f92f00a1a10 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.488 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.470+0000 7f92e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:28.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.542+0000 7f92f5555640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "be602c68-218b-4573-8d19-eef568af9d4e", "id": 1}] 2024-09-15T20:31:28.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.542+0000 7f92f5555640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "be602c68-218b-4573-8d19-eef568af9d4e", "id": 1}] 2024-09-15T20:31:28.547 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.542+0000 7f92f5555640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.548 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.542+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "be602c68-218b-4573-8d19-eef568af9d4e", "id": 1} v 0) -- 0x7f92d4005380 con 0x7f92f00a1a10 2024-09-15T20:31:28.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92e77fe640 1 -- 172.21.15.146:0/1356466764 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "be602c68-218b-4573-8d19-eef568af9d4e", "id": 1}]=0 v4) ==== 112+0+2 (unknown 3167958874 0 940483276) 0x7f92e00069c0 con 0x7f92f00a1a10 2024-09-15T20:31:28.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92e77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "be602c68-218b-4573-8d19-eef568af9d4e", "id": 1}] 2024-09-15T20:31:28.553 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:28.553 INFO:teuthology.orchestra.run.smithi146.stdout:1 2024-09-15T20:31:28.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 10 librados: watch_flush enter 2024-09-15T20:31:28.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 10 librados: watch_flush exit 2024-09-15T20:31:28.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 20 client.4117.objecter shutdown clearing up homeless session... 2024-09-15T20:31:28.554 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 10 client.4117.objecter successfully canceled tick 2024-09-15T20:31:28.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 10 monclient: shutdown 2024-09-15T20:31:28.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:28.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 >> v1:172.21.15.146:6789/0 conn(0x7f92f00a1a10 legacy=0x7f92f0072f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 shutdown_connections 2024-09-15T20:31:28.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 >> 172.21.15.146:0/1356466764 conn(0x7f92f00a3ce0 msgr2=0x7f92f00689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:28.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 shutdown_connections 2024-09-15T20:31:28.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 1 -- 172.21.15.146:0/1356466764 wait complete. 2024-09-15T20:31:28.555 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.550+0000 7f92f5555640 1 librados: shutdown 2024-09-15T20:31:28.570 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 1fab92e2-4880-4f66-86f9-1c59752850a1 2 2024-09-15T20:31:28.689 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient: get_monmap_and_config 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient: build_initial_monmap 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient: monmap: 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:28.688534+0000 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:28.688534+0000 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:28.690 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 1 Processor -- start 2024-09-15T20:31:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 1 -- start start 2024-09-15T20:31:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient: init 2024-09-15T20:31:28.697 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient(hunting): picked mon.noname-b con 0x7fd74c135c10 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient(hunting): picked mon.noname-a con 0x7fd74c09d440 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient(hunting): picked mon.noname-c con 0x7fd74c13c2b0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74c0a2140 con 0x7fd74c09d440 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74c09e4e0 con 0x7fd74c13c2b0 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74c09e7c0 con 0x7fd74c135c10 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient(hunting): authenticate will time out at 1138.598402s 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74ad9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd74c13c2b0 0x7fd74c13e6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49858/0 (socket says 172.21.15.146:49858) 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74ad9d640 1 -- 172.21.15.146:0/2723141847 learned_addr learned my addr 172.21.15.146:0/2723141847 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3762516842 0 0) 0x7fd74c09e4e0 con 0x7fd74c13c2b0 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.698 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient(hunting): my global_id is 4120 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd734003530 con 0x7fd74c13c2b0 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 219627611 0 0) 0x7fd734003530 con 0x7fd74c13c2b0 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 >> v1:172.21.15.78:6790/0 conn(0x7fd74c135c10 legacy=0x7fd74c135ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 >> v1:172.21.15.78:6789/0 conn(0x7fd74c09d440 legacy=0x7fd74c13ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: _finish_hunting 0 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 monclient: found mon.noname-c 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74c09eaa0 con 0x7fd74c13c2b0 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: _finish_auth 0 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: _check_auth_tickets 2024-09-15T20:31:28.699 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd744001a20 con 0x7fd74c13c2b0 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: dump: 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.700 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 5 monclient: authenticate success, global_id 4120 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd744001c40 con 0x7fd74c13c2b0 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd750ed0640 4 set_mon_vals no callback set 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 1 -- 172.21.15.146:0/2723141847 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd744002010 con 0x7fd74c13c2b0 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.686+0000 7fd74959a640 10 monclient: dump: 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.701 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: shutdown 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- 172.21.15.146:0/2723141847 >> v1:172.21.15.146:6789/0 conn(0x7fd74c13c2b0 legacy=0x7fd74c13e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- 172.21.15.146:0/2723141847 shutdown_connections 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- 172.21.15.146:0/2723141847 >> 172.21.15.146:0/2723141847 conn(0x7fd74c0a4b90 msgr2=0x7fd74c0a4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- 172.21.15.146:0/2723141847 shutdown_connections 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- 172.21.15.146:0/2723141847 wait complete. 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: build_initial_monmap 2024-09-15T20:31:28.702 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: monmap: 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 librados: starting msgr at 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 librados: starting objecter 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 Processor -- start 2024-09-15T20:31:28.703 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- start start 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 librados: setting wanted keys 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 librados: calling monclient init 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: init 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient(hunting): picked mon.noname-a con 0x7fd74c135c10 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient(hunting): picked mon.noname-c con 0x7fd74c13c2b0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient(hunting): picked mon.noname-b con 0x7fd74c074690 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74c137720 con 0x7fd74c135c10 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74c137ad0 con 0x7fd74c074690 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd74c074b00 con 0x7fd74c13c2b0 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:28.704 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient(hunting): authenticate will time out at 1138.601790s 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd74ad9d640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd74c074690 0x7fd74c1d2d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49866/0 (socket says 172.21.15.146:49866) 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd74ad9d640 1 -- 172.21.15.146:0/2901536126 learned_addr learned my addr 172.21.15.146:0/2901536126 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 client.?.objecter ms_handle_connect 0x7fd74c074690 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3022415562 0 0) 0x7fd74c137ad0 con 0x7fd74c074690 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): my global_id is 4123 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd72c004ec0 con 0x7fd74c074690 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 client.?.objecter ms_handle_connect 0x7fd74c13c2b0 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 client.?.objecter ms_handle_connect 0x7fd74c135c10 2024-09-15T20:31:28.705 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3714536610 0 0) 0x7fd72c004ec0 con 0x7fd74c074690 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd74c137ad0 con 0x7fd74c074690 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd744001f90 con 0x7fd74c074690 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): dump: 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.706 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1618190650 0 0) 0x7fd74c137720 con 0x7fd74c135c10 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): my global_id is 4119 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd72c004ec0 con 0x7fd74c135c10 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3651440184 0 0) 0x7fd74c074b00 con 0x7fd74c13c2b0 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.707 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient(hunting): my global_id is 4121 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd74c137720 con 0x7fd74c13c2b0 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2307715906 0 0) 0x7fd74c137ad0 con 0x7fd74c074690 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 >> v1:172.21.15.78:6790/0 conn(0x7fd74c13c2b0 legacy=0x7fd74c137010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 >> v1:172.21.15.78:6789/0 conn(0x7fd74c135c10 legacy=0x7fd74c136900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 monclient: found mon.b 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd74c139590 con 0x7fd74c074690 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient: _finish_auth 0 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 5 monclient: authenticate success, global_id 4123 2024-09-15T20:31:28.708 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: _renew_subs 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd74c139870 con 0x7fd74c074690 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: _renew_subs 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd74c1d4420 con 0x7fd74c074690 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 librados: init done 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd744004f80 con 0x7fd74c074690 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd744001530 con 0x7fd74c074690 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd748d99640 4 set_mon_vals no callback set 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 monclient: dump: 2024-09-15T20:31:28.709 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7fd7440054c0 con 0x7fd74c074690 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (unknown 698815671 0 0) 0x7fd744006c10 con 0x7fd74c074690 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 10 client.4123.objecter ms_dispatch 0x7fd74c000ba0 osd_map(4..4 src has 1..4) 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-09-15T20:31:28.710 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-09-15T20:31:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd73affd640 20 client.4123.objecter dump_active .. 0 homeless 2024-09-15T20:31:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.690+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd730005090 con 0x7fd74c074690 2024-09-15T20:31:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.694+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7fd7440067f0 con 0x7fd74c074690 2024-09-15T20:31:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.694+0000 7fd73affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.711 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.694+0000 7fd73affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:28.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.782+0000 7fd750ed0640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1fab92e2-4880-4f66-86f9-1c59752850a1", "id": 2}] 2024-09-15T20:31:28.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.782+0000 7fd750ed0640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1fab92e2-4880-4f66-86f9-1c59752850a1", "id": 2}] 2024-09-15T20:31:28.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.782+0000 7fd750ed0640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.785 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.782+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1fab92e2-4880-4f66-86f9-1c59752850a1", "id": 2} v 0) -- 0x7fd730005380 con 0x7fd74c074690 2024-09-15T20:31:28.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.786+0000 7fd73affd640 1 -- 172.21.15.146:0/2901536126 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1fab92e2-4880-4f66-86f9-1c59752850a1", "id": 2}]=0 v5) ==== 112+0+2 (unknown 1669958770 0 216599125) 0x7fd7440059e0 con 0x7fd74c074690 2024-09-15T20:31:28.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.786+0000 7fd73affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1fab92e2-4880-4f66-86f9-1c59752850a1", "id": 2}] 2024-09-15T20:31:28.792 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.786+0000 7fd73affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:28.793 INFO:teuthology.orchestra.run.smithi146.stdout:2 2024-09-15T20:31:28.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 10 librados: watch_flush enter 2024-09-15T20:31:28.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 10 librados: watch_flush exit 2024-09-15T20:31:28.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 20 client.4123.objecter shutdown clearing up homeless session... 2024-09-15T20:31:28.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 10 client.4123.objecter successfully canceled tick 2024-09-15T20:31:28.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 10 monclient: shutdown 2024-09-15T20:31:28.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:28.794 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 >> v1:172.21.15.146:6789/0 conn(0x7fd74c074690 legacy=0x7fd74c1d2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 shutdown_connections 2024-09-15T20:31:28.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 >> 172.21.15.146:0/2901536126 conn(0x7fd74c0a4b90 msgr2=0x7fd74c0a3470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:28.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 shutdown_connections 2024-09-15T20:31:28.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 1 -- 172.21.15.146:0/2901536126 wait complete. 2024-09-15T20:31:28.795 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.790+0000 7fd750ed0640 1 librados: shutdown 2024-09-15T20:31:28.809 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 74c2a9ff-0635-4664-9dc5-d156f4b60bad 3 2024-09-15T20:31:28.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient: get_monmap_and_config 2024-09-15T20:31:28.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient: build_initial_monmap 2024-09-15T20:31:28.922 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient: monmap: 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:28.921371+0000 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:28.921371+0000 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:28.923 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 1 Processor -- start 2024-09-15T20:31:28.930 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 1 -- start start 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient: init 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe0137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe00a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe013e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe00a6420 con 0x7f1fe0137ce0 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe00a6700 con 0x7f1fe013e5f0 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe00a2110 con 0x7f1fe00a9610 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe64fe640 10 monclient(hunting): authenticate will time out at 1138.831641s 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe4a9b640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1fe013e5f0 0x7f1fe01409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49874/0 (socket says 172.21.15.146:49874) 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.918+0000 7f1fe4a9b640 1 -- 172.21.15.146:0/4075331034 learned_addr learned my addr 172.21.15.146:0/4075331034 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3290604334 0 0) 0x7f1fe00a6700 con 0x7f1fe013e5f0 2024-09-15T20:31:28.931 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient(hunting): my global_id is 4126 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fc8003530 con 0x7f1fe013e5f0 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3576277088 0 0) 0x7f1fc8003530 con 0x7f1fe013e5f0 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 >> v1:172.21.15.78:6790/0 conn(0x7f1fe00a9610 legacy=0x7f1fe013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 >> v1:172.21.15.78:6789/0 conn(0x7f1fe0137ce0 legacy=0x7f1fe01380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 monclient: found mon.noname-c 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.932 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe00a4bc0 con 0x7f1fe013e5f0 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: _finish_auth 0 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1fd8001a20 con 0x7f1fe013e5f0 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: dump: 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.933 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 5 monclient: authenticate success, global_id 4126 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fd8001c40 con 0x7f1fe013e5f0 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 1 -- 172.21.15.146:0/4075331034 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1fd8002050 con 0x7f1fe013e5f0 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 4 set_mon_vals no callback set 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:28.934 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdeffd640 10 monclient: dump: 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: shutdown 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- 172.21.15.146:0/4075331034 >> v1:172.21.15.146:6789/0 conn(0x7f1fe013e5f0 legacy=0x7f1fe01409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- 172.21.15.146:0/4075331034 shutdown_connections 2024-09-15T20:31:28.935 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- 172.21.15.146:0/4075331034 >> 172.21.15.146:0/4075331034 conn(0x7f1fe00aa6c0 msgr2=0x7f1fe00aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- 172.21.15.146:0/4075331034 shutdown_connections 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- 172.21.15.146:0/4075331034 wait complete. 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: build_initial_monmap 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: monmap: 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.936 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 librados: starting msgr at 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 librados: starting objecter 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 Processor -- start 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- start start 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 librados: setting wanted keys 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 librados: calling monclient init 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: init 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient(hunting): picked mon.noname-b con 0x7f1fe00a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient(hunting): picked mon.noname-c con 0x7f1fe0137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient(hunting): picked mon.noname-a con 0x7f1fe013e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe00a0500 con 0x7f1fe013e5f0 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe013e2c0 con 0x7f1fe00a9610 2024-09-15T20:31:28.937 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1fe00a14b0 con 0x7f1fe0137ce0 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient(hunting): authenticate will time out at 1138.834295s 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f1fe00a9610 0x7f1fe009ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49876/0 (socket says 172.21.15.146:49876) 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdffff640 1 -- 172.21.15.146:0/1471733615 learned_addr learned my addr 172.21.15.146:0/1471733615 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 client.?.objecter ms_handle_connect 0x7f1fe00a9610 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 234521832 0 0) 0x7f1fe013e2c0 con 0x7f1fe00a9610 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): my global_id is 4129 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fcc004ec0 con 0x7f1fe00a9610 2024-09-15T20:31:28.938 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 client.?.objecter ms_handle_connect 0x7f1fe013e5f0 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 client.?.objecter ms_handle_connect 0x7f1fe0137ce0 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 152731472 0 0) 0x7f1fcc004ec0 con 0x7f1fe00a9610 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1fe013e2c0 con 0x7f1fe00a9610 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1fd0001a40 con 0x7f1fe00a9610 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): dump: 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.939 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3944855718 0 0) 0x7f1fe00a0500 con 0x7f1fe013e5f0 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient(hunting): my global_id is 4125 2024-09-15T20:31:28.940 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fcc004ec0 con 0x7f1fe013e5f0 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1069024039 0 0) 0x7f1fe013e2c0 con 0x7f1fe00a9610 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 >> v1:172.21.15.78:6790/0 conn(0x7f1fe0137ce0 legacy=0x7f1fe009d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 >> v1:172.21.15.78:6789/0 conn(0x7f1fe013e5f0 legacy=0x7f1fe01d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient: _finish_hunting 0 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 monclient: found mon.b 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fe009efe0 con 0x7f1fe00a9610 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient: _finish_auth 0 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient: _check_auth_tickets 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 5 monclient: authenticate success, global_id 4129 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: _renew_subs 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.941 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1fe00a0220 con 0x7f1fe00a9610 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: _renew_subs 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1fe00a07e0 con 0x7f1fe00a9610 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 librados: init done 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1fd0001d00 con 0x7f1fe00a9610 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fde7fc640 4 set_mon_vals no callback set 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1fd0003a70 con 0x7f1fe00a9610 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 monclient: dump: 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:28.942 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f1fd0004060 con 0x7f1fe00a9610 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (unknown 2957153972 0 0) 0x7f1fd0006820 con 0x7f1fe00a9610 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 10 client.4129.objecter ms_dispatch 0x7f1fe0000ba0 osd_map(5..5 src has 1..5) 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fdcff9640 20 client.4129.objecter dump_active .. 0 homeless 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.943 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:28.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.922+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fc0005090 con 0x7f1fe00a9610 2024-09-15T20:31:28.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.926+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f1fd00064a0 con 0x7f1fe00a9610 2024-09-15T20:31:28.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.926+0000 7f1fdcff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:28.944 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.926+0000 7f1fdcff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:29.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.994+0000 7f1fe64fe640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "74c2a9ff-0635-4664-9dc5-d156f4b60bad", "id": 3}] 2024-09-15T20:31:29.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.994+0000 7f1fe64fe640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "74c2a9ff-0635-4664-9dc5-d156f4b60bad", "id": 3}] 2024-09-15T20:31:29.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.994+0000 7f1fe64fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.000 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:28.994+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "74c2a9ff-0635-4664-9dc5-d156f4b60bad", "id": 3} v 0) -- 0x7f1fc0005380 con 0x7f1fe00a9610 2024-09-15T20:31:29.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fdcff9640 1 -- 172.21.15.146:0/1471733615 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "74c2a9ff-0635-4664-9dc5-d156f4b60bad", "id": 3}]=0 v6) ==== 112+0+2 (unknown 2083681905 0 525046306) 0x7f1fd0006af0 con 0x7f1fe00a9610 2024-09-15T20:31:29.010 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fdcff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "74c2a9ff-0635-4664-9dc5-d156f4b60bad", "id": 3}] 2024-09-15T20:31:29.011 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fdcff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:29.011 INFO:teuthology.orchestra.run.smithi146.stdout:3 2024-09-15T20:31:29.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 10 librados: watch_flush enter 2024-09-15T20:31:29.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 10 librados: watch_flush exit 2024-09-15T20:31:29.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 20 client.4129.objecter shutdown clearing up homeless session... 2024-09-15T20:31:29.012 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 10 client.4129.objecter successfully canceled tick 2024-09-15T20:31:29.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 10 monclient: shutdown 2024-09-15T20:31:29.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:29.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 >> v1:172.21.15.146:6789/0 conn(0x7f1fe00a9610 legacy=0x7f1fe009ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 shutdown_connections 2024-09-15T20:31:29.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 >> 172.21.15.146:0/1471733615 conn(0x7f1fe00aa6c0 msgr2=0x7f1fe00a2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:29.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 shutdown_connections 2024-09-15T20:31:29.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 1 -- 172.21.15.146:0/1471733615 wait complete. 2024-09-15T20:31:29.013 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.006+0000 7f1fe64fe640 1 librados: shutdown 2024-09-15T20:31:29.027 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 0abc805d-20bc-40e6-a25a-0d9694ace9eb 4 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient: get_monmap_and_config 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient: build_initial_monmap 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient: monmap: 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:29.141829+0000 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:29.141829+0000 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:29.143 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:29.144 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.150 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 1 Processor -- start 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 1 -- start start 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient: init 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient(hunting): picked mon.noname-c con 0x7f65a0134c20 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient(hunting): picked mon.noname-b con 0x7f65a009fc80 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient(hunting): picked mon.noname-a con 0x7f65a013b1a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a009efa0 con 0x7f65a013b1a0 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a009e4a0 con 0x7f65a0134c20 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a0074fa0 con 0x7f65a009fc80 2024-09-15T20:31:29.151 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a8069640 10 monclient(hunting): authenticate will time out at 1139.051756s 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a5e05640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f65a0134c20 0x7f65a0135000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49888/0 (socket says 172.21.15.146:49888) 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.138+0000 7f65a5e05640 1 -- 172.21.15.146:0/3253205067 learned_addr learned my addr 172.21.15.146:0/3253205067 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 12718179 0 0) 0x7f65a009e4a0 con 0x7f65a0134c20 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient(hunting): my global_id is 4132 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f658c003530 con 0x7f65a0134c20 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 30689680 0 0) 0x7f658c003530 con 0x7f65a0134c20 2024-09-15T20:31:29.152 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 >> v1:172.21.15.78:6790/0 conn(0x7f65a009fc80 legacy=0x7f65a0139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 >> v1:172.21.15.78:6789/0 conn(0x7f65a013b1a0 legacy=0x7f65a013d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: _finish_hunting 0 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 monclient: found mon.noname-c 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a009f280 con 0x7f65a0134c20 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: _finish_auth 0 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: _check_auth_tickets 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65980016a0 con 0x7f65a0134c20 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 5 monclient: authenticate success, global_id 4132 2024-09-15T20:31:29.153 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: dump: 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.154 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6598001840 con 0x7f65a0134c20 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 4 set_mon_vals no callback set 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 1 -- 172.21.15.146:0/3253205067 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6598001c50 con 0x7f65a0134c20 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a4e03640 10 monclient: dump: 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.155 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: shutdown 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3253205067 >> v1:172.21.15.146:6789/0 conn(0x7f65a0134c20 legacy=0x7f65a0135000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3253205067 shutdown_connections 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3253205067 >> 172.21.15.146:0/3253205067 conn(0x7f65a00a3aa0 msgr2=0x7f65a00a3eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3253205067 shutdown_connections 2024-09-15T20:31:29.156 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3253205067 wait complete. 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: build_initial_monmap 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: monmap: 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 librados: starting msgr at 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 librados: starting objecter 2024-09-15T20:31:29.157 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 Processor -- start 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- start start 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 librados: setting wanted keys 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 librados: calling monclient init 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: init 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient(hunting): picked mon.noname-b con 0x7f65a009fc80 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient(hunting): picked mon.noname-a con 0x7f65a0134c20 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient(hunting): picked mon.noname-c con 0x7f65a013b1a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a0133990 con 0x7f65a0134c20 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a5e05640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f65a009fc80 0x7f65a013b580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49900/0 (socket says 172.21.15.146:49900) 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a5e05640 1 -- 172.21.15.146:0/3280952160 learned_addr learned my addr 172.21.15.146:0/3280952160 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a013c3a0 con 0x7f65a009fc80 2024-09-15T20:31:29.158 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65a0074930 con 0x7f65a013b1a0 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient(hunting): authenticate will time out at 1139.055118s 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 client.?.objecter ms_handle_connect 0x7f65a009fc80 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2148054651 0 0) 0x7f65a013c3a0 con 0x7f65a009fc80 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): my global_id is 4135 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6584004ec0 con 0x7f65a009fc80 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 client.?.objecter ms_handle_connect 0x7f65a013b1a0 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 client.?.objecter ms_handle_connect 0x7f65a0134c20 2024-09-15T20:31:29.159 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3471331213 0 0) 0x7f6584004ec0 con 0x7f65a009fc80 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65a013c3a0 con 0x7f65a009fc80 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6598001b90 con 0x7f65a009fc80 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): dump: 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.160 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1800723278 0 0) 0x7f65a0074930 con 0x7f65a013b1a0 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): my global_id is 4133 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6584004ec0 con 0x7f65a013b1a0 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1531786359 0 0) 0x7f65a0133990 con 0x7f65a0134c20 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.161 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient(hunting): my global_id is 4131 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65a0074930 con 0x7f65a0134c20 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2850983515 0 0) 0x7f65a013c3a0 con 0x7f65a009fc80 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 >> v1:172.21.15.78:6790/0 conn(0x7f65a013b1a0 legacy=0x7f65a01d5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 >> v1:172.21.15.78:6789/0 conn(0x7f65a0134c20 legacy=0x7f65a013bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 monclient: found mon.b 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65a011e1e0 con 0x7f65a009fc80 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:29.162 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 5 monclient: authenticate success, global_id 4135 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: _renew_subs 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65a011d940 con 0x7f65a009fc80 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 client.4135.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: _renew_subs 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65a011df00 con 0x7f65a009fc80 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.142+0000 7f65a8069640 1 librados: init done 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6598001820 con 0x7f65a009fc80 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6598004da0 con 0x7f65a009fc80 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 10 monclient: dump: 2024-09-15T20:31:29.163 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f6597fff640 4 set_mon_vals no callback set 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f65980052d0 con 0x7f65a009fc80 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (unknown 2021294381 0 0) 0x7f65980068d0 con 0x7f65a009fc80 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 10 client.4135.objecter ms_dispatch 0x7f65a0000ba0 osd_map(6..6 src has 1..6) 2024-09-15T20:31:29.164 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 3 client.4135.objecter handle_osd_map got epochs [6,6] > 0 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 3 client.4135.objecter handle_osd_map decoding full epoch 6 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 20 client.4135.objecter dump_active .. 0 homeless 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65a8069640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65a8069640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65a8069640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6588005090 con 0x7f65a009fc80 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f6598006ba0 con 0x7f65a009fc80 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.165 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.146+0000 7f65967fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:29.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.218+0000 7f65a8069640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "0abc805d-20bc-40e6-a25a-0d9694ace9eb", "id": 4}] 2024-09-15T20:31:29.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.218+0000 7f65a8069640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "0abc805d-20bc-40e6-a25a-0d9694ace9eb", "id": 4}] 2024-09-15T20:31:29.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.218+0000 7f65a8069640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.221 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.218+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "0abc805d-20bc-40e6-a25a-0d9694ace9eb", "id": 4} v 0) -- 0x7f6588005380 con 0x7f65a009fc80 2024-09-15T20:31:29.231 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.226+0000 7f65967fc640 1 -- 172.21.15.146:0/3280952160 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "0abc805d-20bc-40e6-a25a-0d9694ace9eb", "id": 4}]=0 v7) ==== 112+0+2 (unknown 3555891857 0 1697012583) 0x7f6598006540 con 0x7f65a009fc80 2024-09-15T20:31:29.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.226+0000 7f65967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "0abc805d-20bc-40e6-a25a-0d9694ace9eb", "id": 4}] 2024-09-15T20:31:29.232 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.226+0000 7f65967fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:29.232 INFO:teuthology.orchestra.run.smithi146.stdout:4 2024-09-15T20:31:29.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 10 librados: watch_flush enter 2024-09-15T20:31:29.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 10 librados: watch_flush exit 2024-09-15T20:31:29.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 20 client.4135.objecter shutdown clearing up homeless session... 2024-09-15T20:31:29.233 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 10 client.4135.objecter successfully canceled tick 2024-09-15T20:31:29.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 10 monclient: shutdown 2024-09-15T20:31:29.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:29.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 >> v1:172.21.15.146:6789/0 conn(0x7f65a009fc80 legacy=0x7f65a013b580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 shutdown_connections 2024-09-15T20:31:29.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 >> 172.21.15.146:0/3280952160 conn(0x7f65a00a3aa0 msgr2=0x7f65a00689d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:29.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 shutdown_connections 2024-09-15T20:31:29.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 1 -- 172.21.15.146:0/3280952160 wait complete. 2024-09-15T20:31:29.234 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.230+0000 7f65a8069640 1 librados: shutdown 2024-09-15T20:31:29.248 DEBUG:teuthology.orchestra.run.smithi146:> sudo ceph --cluster ceph osd new 08c34b4f-deb1-401a-aab7-00854ae6712f 5 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient: get_monmap_and_config 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient: build_initial_monmap 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient: monmap: 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:29.356869+0000 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:29.356869+0000 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:29.358 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.359 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:29.359 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:29.359 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:29.359 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 1 Processor -- start 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 1 -- start start 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient: init 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5cd0137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5cd00a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f5cd013e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd00a6420 con 0x7f5cd0137ce0 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd00a6700 con 0x7f5cd013e5f0 2024-09-15T20:31:29.362 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd00a2110 con 0x7f5cd00a9610 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient(hunting): authenticate will time out at 1139.266880s 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd50ba640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5cd013e5f0 0x7f5cd01409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49908/0 (socket says 172.21.15.146:49908) 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd50ba640 1 -- 172.21.15.146:0/920458798 learned_addr learned my addr 172.21.15.146:0/920458798 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399791187 0 0) 0x7f5cd00a6700 con 0x7f5cd013e5f0 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient(hunting): my global_id is 4138 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cc0003530 con 0x7f5cd013e5f0 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 548311179 0 0) 0x7f5cc0003530 con 0x7f5cd013e5f0 2024-09-15T20:31:29.363 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 >> v1:172.21.15.78:6790/0 conn(0x7f5cd00a9610 legacy=0x7f5cd013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 >> v1:172.21.15.78:6789/0 conn(0x7f5cd0137ce0 legacy=0x7f5cd01380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 monclient: found mon.noname-c 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd00a4bc0 con 0x7f5cd013e5f0 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5cc8001a20 con 0x7f5cd013e5f0 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:29.364 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: dump: 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 5 monclient: authenticate success, global_id 4138 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:29.365 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5cc8001bc0 con 0x7f5cd013e5f0 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5cd6b1d640 4 set_mon_vals no callback set 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 1 -- 172.21.15.146:0/920458798 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5cc8001fd0 con 0x7f5cd013e5f0 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.354+0000 7f5ccf7fe640 10 monclient: dump: 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.366 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: shutdown 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/920458798 >> v1:172.21.15.146:6789/0 conn(0x7f5cd013e5f0 legacy=0x7f5cd01409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/920458798 shutdown_connections 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/920458798 >> 172.21.15.146:0/920458798 conn(0x7f5cd00aa6c0 msgr2=0x7f5cd00aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/920458798 shutdown_connections 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/920458798 wait complete. 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: build_initial_monmap 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: monmap: 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 0 2024-09-15T20:31:29.367 INFO:teuthology.orchestra.run.smithi146.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 0.000000 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:created 0.000000 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 librados: starting msgr at 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 librados: starting objecter 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 Processor -- start 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- start start 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 librados: setting wanted keys 2024-09-15T20:31:29.368 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 librados: calling monclient init 2024-09-15T20:31:29.369 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: init 2024-09-15T20:31:29.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:29.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:29.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient(hunting): picked mon.noname-a con 0x7f5cd00a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:29.371 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient(hunting): picked mon.noname-b con 0x7f5cd0137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient(hunting): picked mon.noname-c con 0x7f5cd013e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd01372d0 con 0x7f5cd00a9610 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd009e2f0 con 0x7f5cd0137ce0 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cd00a0650 con 0x7f5cd013e5f0 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient(hunting): authenticate will time out at 1139.269922s 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccffff640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f5cd0137ce0 0x7f5cd009d130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:49914/0 (socket says 172.21.15.146:49914) 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccffff640 1 -- 172.21.15.146:0/207041212 learned_addr learned my addr 172.21.15.146:0/207041212 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 client.?.objecter ms_handle_connect 0x7f5cd0137ce0 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682676627 0 0) 0x7f5cd009e2f0 con 0x7f5cd0137ce0 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.372 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient(hunting): my global_id is 4141 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cb4004ec0 con 0x7f5cd0137ce0 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 client.?.objecter ms_handle_connect 0x7f5cd00a9610 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 client.?.objecter ms_handle_connect 0x7f5cd013e5f0 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2429878327 0 0) 0x7f5cb4004ec0 con 0x7f5cd0137ce0 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5cd009e2f0 con 0x7f5cd0137ce0 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5cbc001a40 con 0x7f5cd0137ce0 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient(hunting): dump: 2024-09-15T20:31:29.373 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4275516980 0 0) 0x7f5cd009e2f0 con 0x7f5cd0137ce0 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 >> v1:172.21.15.78:6790/0 conn(0x7f5cd013e5f0 legacy=0x7f5cd009d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 >> v1:172.21.15.78:6789/0 conn(0x7f5cd00a9610 legacy=0x7f5cd009ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.374 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 monclient: found mon.b 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cd009fdb0 con 0x7f5cd0137ce0 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 5 monclient: authenticate success, global_id 4141 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: _renew_subs 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5cd013de90 con 0x7f5cd0137ce0 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:29.375 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: _renew_subs 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5cd01d6130 con 0x7f5cd0137ce0 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 librados: init done 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5cbc001d00 con 0x7f5cd0137ce0 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5cbc003a70 con 0x7f5cd0137ce0 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cce7fc640 4 set_mon_vals no callback set 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 monclient: dump: 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:epoch 1 2024-09-15T20:31:29.376 INFO:teuthology.orchestra.run.smithi146.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:election_strategy: 1 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f5cbc004060 con 0x7f5cd0137ce0 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (unknown 604414003 0 0) 0x7f5cbc006a80 con 0x7f5cd0137ce0 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 10 client.4141.objecter ms_dispatch 0x7f5cd0000ba0 osd_map(7..7 src has 1..7) 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-09-15T20:31:29.377 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-09-15T20:31:29.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5ccd7fa640 20 client.4141.objecter dump_active .. 0 homeless 2024-09-15T20:31:29.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.358+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cb0005090 con 0x7f5cd0137ce0 2024-09-15T20:31:29.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.362+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f5cbc005d30 con 0x7f5cd0137ce0 2024-09-15T20:31:29.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.362+0000 7f5ccd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.378 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.362+0000 7f5ccd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:29.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.430+0000 7f5cd6b1d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "08c34b4f-deb1-401a-aab7-00854ae6712f", "id": 5}] 2024-09-15T20:31:29.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.430+0000 7f5cd6b1d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "08c34b4f-deb1-401a-aab7-00854ae6712f", "id": 5}] 2024-09-15T20:31:29.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.430+0000 7f5cd6b1d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:29.436 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.430+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd new", "uuid": "08c34b4f-deb1-401a-aab7-00854ae6712f", "id": 5} v 0) -- 0x7f5cb0005380 con 0x7f5cd0137ce0 2024-09-15T20:31:29.445 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5ccd7fa640 1 -- 172.21.15.146:0/207041212 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "08c34b4f-deb1-401a-aab7-00854ae6712f", "id": 5}]=0 v8) ==== 112+0+2 (unknown 970472320 0 1988412176) 0x7f5cbc0057d0 con 0x7f5cd0137ce0 2024-09-15T20:31:29.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5ccd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "08c34b4f-deb1-401a-aab7-00854ae6712f", "id": 5}] 2024-09-15T20:31:29.446 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5ccd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:29.446 INFO:teuthology.orchestra.run.smithi146.stdout:5 2024-09-15T20:31:29.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 10 librados: watch_flush enter 2024-09-15T20:31:29.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 10 librados: watch_flush exit 2024-09-15T20:31:29.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 20 client.4141.objecter shutdown clearing up homeless session... 2024-09-15T20:31:29.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 10 client.4141.objecter successfully canceled tick 2024-09-15T20:31:29.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 10 monclient: shutdown 2024-09-15T20:31:29.447 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:29.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 >> v1:172.21.15.146:6789/0 conn(0x7f5cd0137ce0 legacy=0x7f5cd009d130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 shutdown_connections 2024-09-15T20:31:29.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 >> 172.21.15.146:0/207041212 conn(0x7f5cd00aa6c0 msgr2=0x7f5cd00a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:29.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 shutdown_connections 2024-09-15T20:31:29.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 1 -- 172.21.15.146:0/207041212 wait complete. 2024-09-15T20:31:29.448 INFO:teuthology.orchestra.run.smithi146.stderr:2024-09-15T20:31:29.442+0000 7f5cd6b1d640 1 librados: shutdown 2024-09-15T20:31:29.461 INFO:tasks.ceph.osd.0:Restarting daemon 2024-09-15T20:31:29.461 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-09-15T20:31:29.464 INFO:tasks.ceph.osd.0:Started 2024-09-15T20:31:29.464 INFO:tasks.ceph.osd.1:Restarting daemon 2024-09-15T20:31:29.464 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-09-15T20:31:29.467 INFO:tasks.ceph.osd.1:Started 2024-09-15T20:31:29.467 INFO:tasks.ceph.osd.2:Restarting daemon 2024-09-15T20:31:29.467 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-09-15T20:31:29.469 INFO:tasks.ceph.osd.2:Started 2024-09-15T20:31:29.469 INFO:tasks.ceph.osd.3:Restarting daemon 2024-09-15T20:31:29.469 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-09-15T20:31:29.471 INFO:tasks.ceph.osd.3:Started 2024-09-15T20:31:29.471 INFO:tasks.ceph.osd.4:Restarting daemon 2024-09-15T20:31:29.471 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-09-15T20:31:29.473 INFO:tasks.ceph.osd.4:Started 2024-09-15T20:31:29.473 INFO:tasks.ceph.osd.5:Restarting daemon 2024-09-15T20:31:29.473 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-09-15T20:31:29.475 INFO:tasks.ceph.osd.5:Started 2024-09-15T20:31:29.475 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:31:29.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.592+0000 7f22ee0dc640 10 monclient: get_monmap_and_config 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.592+0000 7f22ee0dc640 10 monclient: build_initial_monmap 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.592+0000 7f22ee0dc640 10 monclient: monmap: 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:29.596517+0000 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:29.596517+0000 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:29.598 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:29.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 Processor -- start 2024-09-15T20:31:29.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- start start 2024-09-15T20:31:29.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: init 2024-09-15T20:31:29.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:29.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient(hunting): picked mon.noname-b con 0x7f22e8137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient(hunting): picked mon.noname-a con 0x7f22e80a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient(hunting): picked mon.noname-c con 0x7f22e813e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a6420 con 0x7f22e80a9610 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a6700 con 0x7f22e813e5f0 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a2110 con 0x7f22e8137ce0 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient(hunting): authenticate will time out at 1150.481016s 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e77fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f22e8137ce0 0x7f22e81380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43782/0 (socket says 172.21.15.78:43782) 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e77fe640 1 -- 172.21.15.78:0/1740167821 learned_addr learned my addr 172.21.15.78:0/1740167821 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2873110258 0 0) 0x7f22e80a6420 con 0x7f22e80a9610 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient(hunting): my global_id is 4158 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d0003530 con 0x7f22e80a9610 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 24682884 0 0) 0x7f22e80a2110 con 0x7f22e8137ce0 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient(hunting): my global_id is 4160 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e80a6420 con 0x7f22e8137ce0 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2804087628 0 0) 0x7f22e80a6420 con 0x7f22e8137ce0 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 >> v1:172.21.15.146:6789/0 conn(0x7f22e813e5f0 legacy=0x7f22e81409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 >> v1:172.21.15.78:6789/0 conn(0x7f22e80a9610 legacy=0x7f22e813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 monclient: found mon.noname-b 2024-09-15T20:31:29.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e80a4bc0 con 0x7f22e8137ce0 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f22dc001a00 con 0x7f22e8137ce0 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: dump: 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.608 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 5 monclient: authenticate success, global_id 4160 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22dc001ba0 con 0x7f22e8137ce0 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 1 -- 172.21.15.78:0/1740167821 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f22dc001fb0 con 0x7f22e8137ce0 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 4 set_mon_vals no callback set 2024-09-15T20:31:29.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22e67fc640 10 monclient: dump: 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: shutdown 2024-09-15T20:31:29.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- 172.21.15.78:0/1740167821 >> v1:172.21.15.78:6790/0 conn(0x7f22e8137ce0 legacy=0x7f22e81380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- 172.21.15.78:0/1740167821 shutdown_connections 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- 172.21.15.78:0/1740167821 >> 172.21.15.78:0/1740167821 conn(0x7f22e80aa6c0 msgr2=0x7f22e80aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- 172.21.15.78:0/1740167821 shutdown_connections 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- 172.21.15.78:0/1740167821 wait complete. 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: build_initial_monmap 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: monmap: 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:29.611 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 librados: starting msgr at 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 librados: starting objecter 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 Processor -- start 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 -- start start 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 librados: setting wanted keys 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 1 librados: calling monclient init 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: init 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:29.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.596+0000 7f22ee0dc640 10 monclient(hunting): picked mon.noname-b con 0x7f22e80a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient(hunting): picked mon.noname-c con 0x7f22e8137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient(hunting): picked mon.noname-a con 0x7f22e813e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a92e0 con 0x7f22e813e5f0 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e80a14b0 con 0x7f22e80a9610 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22e813bc50 con 0x7f22e8137ce0 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient(hunting): authenticate will time out at 1150.484071s 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22e6ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f22e8137ce0 0x7f22e81d1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43784/0 (socket says 172.21.15.78:43784) 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22e7fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f22e813e5f0 0x7f22e81d5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:39710/0 (socket says 172.21.15.78:39710) 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22e6ffd640 1 -- 172.21.15.78:0/3278138036 learned_addr learned my addr 172.21.15.78:0/3278138036 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 client.?.objecter ms_handle_connect 0x7f22e813e5f0 2024-09-15T20:31:29.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 client.?.objecter ms_handle_connect 0x7f22e8137ce0 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4264677519 0 0) 0x7f22e80a92e0 con 0x7f22e813e5f0 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): my global_id is 4161 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22d4004ef0 con 0x7f22e813e5f0 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 client.?.objecter ms_handle_connect 0x7f22e80a9610 2024-09-15T20:31:29.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704548238 0 0) 0x7f22e813bc50 con 0x7f22e8137ce0 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): my global_id is 4163 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e80a92e0 con 0x7f22e8137ce0 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2379375748 0 0) 0x7f22e80a14b0 con 0x7f22e80a9610 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): my global_id is 4165 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f22e813bc50 con 0x7f22e80a9610 2024-09-15T20:31:29.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3322203267 0 0) 0x7f22d4004ef0 con 0x7f22e813e5f0 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22e80a14b0 con 0x7f22e813e5f0 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f22e0001900 con 0x7f22e813e5f0 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): dump: 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:29.616 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2505586955 0 0) 0x7f22e80a92e0 con 0x7f22e8137ce0 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f22d4004ef0 con 0x7f22e8137ce0 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f22d8001840 con 0x7f22e8137ce0 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient(hunting): dump: 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.617 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3160989614 0 0) 0x7f22e80a14b0 con 0x7f22e813e5f0 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 >> v1:172.21.15.78:6790/0 conn(0x7f22e8137ce0 legacy=0x7f22e81d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 >> v1:172.21.15.146:6789/0 conn(0x7f22e80a9610 legacy=0x7f22e813d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:29.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 monclient: found mon.a 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22e813bf30 con 0x7f22e813e5f0 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient: _finish_auth 0 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 5 monclient: authenticate success, global_id 4161 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient: _renew_subs 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f22e813a1d0 con 0x7f22e813e5f0 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 client.4161.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:29.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient: _renew_subs 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f22e813e030 con 0x7f22e813e5f0 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 1 librados: init done 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f22e0003620 con 0x7f22e813e5f0 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f22e0003970 con 0x7f22e813e5f0 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22e5ffb640 4 set_mon_vals no callback set 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 monclient: dump: 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:29.620 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f22e0003f20 con 0x7f22e813e5f0 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 750923436 0 0) 0x7f22e0004210 con 0x7f22e813e5f0 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 10 client.4161.objecter ms_dispatch 0x7f22e8000ba0 osd_map(8..8 src has 1..8) 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 3 client.4161.objecter handle_osd_map got epochs [8,8] > 0 2024-09-15T20:31:29.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 3 client.4161.objecter handle_osd_map decoding full epoch 8 2024-09-15T20:31:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22cffff640 20 client.4161.objecter dump_active .. 0 homeless 2024-09-15T20:31:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.600+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22c8005090 con 0x7f22e813e5f0 2024-09-15T20:31:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.604+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f22e00057d0 con 0x7f22e813e5f0 2024-09-15T20:31:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.604+0000 7f22cffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:29.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.604+0000 7f22cffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:29.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:29.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:29.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:29.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f22c8005380 con 0x7f22e813e5f0 2024-09-15T20:31:29.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22cffff640 1 -- 172.21.15.78:0/3278138036 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 4082274615) 0x7f22e00059b0 con 0x7f22e813e5f0 2024-09-15T20:31:29.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22cffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:29.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22cffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:29.667 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:29.667 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":8,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","created":"2024-09-15T20:31:27.702188+0000","modified":"2024-09-15T20:31:29.437051+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T20:31:29.437051+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9003b16b-6a87-4659-8436-ff541ea8dfcb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be602c68-218b-4573-8d19-eef568af9d4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1fab92e2-4880-4f66-86f9-1c59752850a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"74c2a9ff-0635-4664-9dc5-d156f4b60bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0abc805d-20bc-40e6-a25a-0d9694ace9eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"08c34b4f-deb1-401a-aab7-00854ae6712f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 10 librados: watch_flush enter 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 10 librados: watch_flush exit 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 20 client.4161.objecter shutdown clearing up homeless session... 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 10 client.4161.objecter successfully canceled tick 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 10 monclient: shutdown 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 >> v1:172.21.15.78:6789/0 conn(0x7f22e813e5f0 legacy=0x7f22e81d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 shutdown_connections 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 >> 172.21.15.78:0/3278138036 conn(0x7f22e80aa6c0 msgr2=0x7f22e8073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 shutdown_connections 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 1 -- 172.21.15.78:0/3278138036 wait complete. 2024-09-15T20:31:29.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:29.664+0000 7f22ee0dc640 1 librados: shutdown 2024-09-15T20:31:29.683 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-09-15T20:31:29.683 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-15T20:31:30.287 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient: get_monmap_and_config 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient: build_initial_monmap 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient: monmap: 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:epoch 0 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:30.407367+0000 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:30.407367+0000 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:30.408 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:30.409 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:30.409 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 1 Processor -- start 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 1 -- start start 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient: init 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient(hunting): picked mon.noname-a con 0x7f4f480a1910 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient(hunting): picked mon.noname-b con 0x7f4f4809e9d0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient(hunting): picked mon.noname-c con 0x7f4f480a2400 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4809ee50 con 0x7f4f480a1910 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4809e790 con 0x7f4f480a2400 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4806e4f0 con 0x7f4f4809e9d0 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:30.412 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.404+0000 7f4f50736640 10 monclient(hunting): authenticate will time out at 1151.291983s 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4e4d2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4f480a1910 0x7f4f480a1cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44526/0 (socket says 172.21.15.78:44526) 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4dcd1640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f4f4809e9d0 0x7f4f48139b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50530/0 (socket says 172.21.15.78:50530) 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4e4d2640 1 -- 172.21.15.78:0/1293438073 learned_addr learned my addr 172.21.15.78:0/1293438073 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1145988364 0 0) 0x7f4f440014a0 con 0x7f4f480a1910 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient(hunting): _negotiate 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient(hunting): my global_id is 4164 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f34003530 con 0x7f4f480a1910 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2028549669 0 0) 0x7f4f4806e4f0 con 0x7f4f4809e9d0 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient(hunting): _negotiate 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:30.413 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient(hunting): my global_id is 4166 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f440014a0 con 0x7f4f4809e9d0 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2232887424 0 0) 0x7f4f34003530 con 0x7f4f480a1910 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 >> v1:172.21.15.78:6790/0 conn(0x7f4f4809e9d0 legacy=0x7f4f48139b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 >> v1:172.21.15.146:6789/0 conn(0x7f4f480a2400 legacy=0x7f4f4813d250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: _finish_hunting 0 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 monclient: found mon.noname-a 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4806e980 con 0x7f4f480a1910 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: _finish_auth 0 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: _check_auth_tickets 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:30.414 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4f44001bf0 con 0x7f4f480a1910 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 5 monclient: authenticate success, global_id 4164 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: dump: 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:30.415 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f44001e10 con 0x7f4f480a1910 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 4 set_mon_vals no callback set 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 1 -- 172.21.15.78:0/1293438073 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4f44004dd0 con 0x7f4f480a1910 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4d4d0640 10 monclient: dump: 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.416 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient: shutdown 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- 172.21.15.78:0/1293438073 >> v1:172.21.15.78:6789/0 conn(0x7f4f480a1910 legacy=0x7f4f480a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- 172.21.15.78:0/1293438073 shutdown_connections 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- 172.21.15.78:0/1293438073 >> 172.21.15.78:0/1293438073 conn(0x7f4f480a6500 msgr2=0x7f4f480a2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- 172.21.15.78:0/1293438073 shutdown_connections 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- 172.21.15.78:0/1293438073 wait complete. 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient: build_initial_monmap 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient: monmap: 2024-09-15T20:31:30.417 INFO:teuthology.misc.health.smithi078.stderr:epoch 0 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:created 0.000000 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 librados: starting msgr at 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 librados: starting objecter 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 Processor -- start 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- start start 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 librados: setting wanted keys 2024-09-15T20:31:30.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 librados: calling monclient init 2024-09-15T20:31:30.419 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient: init 2024-09-15T20:31:30.421 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient(hunting): picked mon.noname-a con 0x7f4f4809e9d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient(hunting): picked mon.noname-b con 0x7f4f480a1910 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient(hunting): picked mon.noname-c con 0x7f4f480a2400 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4811f930 con 0x7f4f4809e9d0 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4811fc10 con 0x7f4f480a1910 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4f4811d370 con 0x7f4f480a2400 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f50736640 10 monclient(hunting): authenticate will time out at 1151.295139s 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4e4d2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4f4809e9d0 0x7f4f4811c550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44538/0 (socket says 172.21.15.78:44538) 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f4e4d2640 1 -- 172.21.15.78:0/400632408 learned_addr learned my addr 172.21.15.78:0/400632408 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 client.?.objecter ms_handle_connect 0x7f4f4809e9d0 2024-09-15T20:31:30.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 client.?.objecter ms_handle_connect 0x7f4f480a2400 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 client.?.objecter ms_handle_connect 0x7f4f480a1910 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2602403493 0 0) 0x7f4f4811f930 con 0x7f4f4809e9d0 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): my global_id is 4167 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f2c004ef0 con 0x7f4f4809e9d0 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1495992567 0 0) 0x7f4f4811d370 con 0x7f4f480a2400 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:30.423 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): my global_id is 4169 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f4811f930 con 0x7f4f480a2400 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3535242851 0 0) 0x7f4f2c004ef0 con 0x7f4f4809e9d0 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f4811d370 con 0x7f4f4809e9d0 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778029895 0 0) 0x7f4f4811fc10 con 0x7f4f480a1910 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): my global_id is 4171 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4f2c004ef0 con 0x7f4f480a1910 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4f44001c70 con 0x7f4f4809e9d0 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): dump: 2024-09-15T20:31:30.424 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2905763389 0 0) 0x7f4f4811f930 con 0x7f4f480a2400 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4f4811fc10 con 0x7f4f480a2400 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4f400017b0 con 0x7f4f480a2400 2024-09-15T20:31:30.425 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.408+0000 7f4f3affd640 10 monclient(hunting): dump: 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1654572209 0 0) 0x7f4f4811d370 con 0x7f4f4809e9d0 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 >> v1:172.21.15.78:6790/0 conn(0x7f4f480a2400 legacy=0x7f4f481d4cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 >> v1:172.21.15.146:6789/0 conn(0x7f4f480a1910 legacy=0x7f4f4811cc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:30.426 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 monclient: found mon.a 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f4811f080 con 0x7f4f4809e9d0 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: _finish_auth 0 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 5 monclient: authenticate success, global_id 4167 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 10 monclient: _renew_subs 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4f4811db90 con 0x7f4f4809e9d0 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 10 client.4167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:30.427 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 10 monclient: _renew_subs 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4f481203f0 con 0x7f4f4809e9d0 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 1 librados: init done 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4f44001e10 con 0x7f4f4809e9d0 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4f44005230 con 0x7f4f4809e9d0 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: dump: 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.428 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 1) ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4f44005760 con 0x7f4f4809e9d0 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (unknown 750923436 0 0) 0x7f4f44006e80 con 0x7f4f4809e9d0 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 client.4167.objecter ms_dispatch 0x7f4f48000ba0 osd_map(8..8 src has 1..8) 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 3 client.4167.objecter handle_osd_map got epochs [8,8] > 0 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 3 client.4167.objecter handle_osd_map decoding full epoch 8 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f4cccf640 4 set_mon_vals no callback set 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 20 client.4167.objecter dump_active .. 0 homeless 2024-09-15T20:31:30.429 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:30.430 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:30.430 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:30.430 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f30005090 con 0x7f4f4809e9d0 2024-09-15T20:31:30.430 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+75856 (unknown 1092875540 0 1702466736) 0x7f4f4809ee50 con 0x7f4f4809e9d0 2024-09-15T20:31:30.430 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:30.430 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.412+0000 7f4f3affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:30.476 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.472+0000 7f4f50736640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:30.476 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.472+0000 7f4f50736640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:30.476 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.472+0000 7f4f50736640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:30.476 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.472+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4f30005380 con 0x7f4f4809e9d0 2024-09-15T20:31:30.477 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.472+0000 7f4f3affd640 1 -- 172.21.15.78:0/400632408 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v8) ==== 74+0+5605 (unknown 150273500 0 4082274615) 0x7f4f440069a0 con 0x7f4f4809e9d0 2024-09-15T20:31:30.477 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.472+0000 7f4f3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:30.477 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.472+0000 7f4f3affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:30.477 INFO:teuthology.misc.health.smithi078.stdout: 2024-09-15T20:31:30.477 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":8,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","created":"2024-09-15T20:31:27.702188+0000","modified":"2024-09-15T20:31:29.437051+0000","last_up_change":"0.000000","last_in_change":"2024-09-15T20:31:29.437051+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9003b16b-6a87-4659-8436-ff541ea8dfcb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be602c68-218b-4573-8d19-eef568af9d4e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1fab92e2-4880-4f66-86f9-1c59752850a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"74c2a9ff-0635-4664-9dc5-d156f4b60bad","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"0abc805d-20bc-40e6-a25a-0d9694ace9eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"08c34b4f-deb1-401a-aab7-00854ae6712f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:31:30.478 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 10 librados: watch_flush enter 2024-09-15T20:31:30.478 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 10 librados: watch_flush exit 2024-09-15T20:31:30.478 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 20 client.4167.objecter shutdown clearing up homeless session... 2024-09-15T20:31:30.478 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 10 client.4167.objecter successfully canceled tick 2024-09-15T20:31:30.478 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 10 monclient: shutdown 2024-09-15T20:31:30.478 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:30.479 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 >> v1:172.21.15.78:6789/0 conn(0x7f4f4809e9d0 legacy=0x7f4f4811c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:30.479 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 shutdown_connections 2024-09-15T20:31:30.479 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 >> 172.21.15.78:0/400632408 conn(0x7f4f480a6500 msgr2=0x7f4f480714c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:30.479 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 shutdown_connections 2024-09-15T20:31:30.479 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 1 -- 172.21.15.78:0/400632408 wait complete. 2024-09-15T20:31:30.479 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:30.476+0000 7f4f50736640 1 librados: shutdown 2024-09-15T20:31:30.493 DEBUG:teuthology.misc:0 of 6 OSDs are up 2024-09-15T20:31:30.574 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-09-15T20:31:30.572+0000 7ff6a45487c0 -1 Falling back to public interface 2024-09-15T20:31:30.582 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-15T20:31:30.578+0000 7fae8c4ff7c0 -1 Falling back to public interface 2024-09-15T20:31:30.708 INFO:tasks.ceph.mgr.x.smithi146.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-15T20:31:30.708 INFO:tasks.ceph.mgr.x.smithi146.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-15T20:31:30.708 INFO:tasks.ceph.mgr.x.smithi146.stderr: from numpy import show_config as show_numpy_config 2024-09-15T20:31:30.823 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-09-15T20:31:30.820+0000 7ff7a09567c0 -1 Falling back to public interface 2024-09-15T20:31:30.838 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-09-15T20:31:30.834+0000 7fbcd83537c0 -1 Falling back to public interface 2024-09-15T20:31:31.077 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T20:31:31.072+0000 7fc120f337c0 -1 Falling back to public interface 2024-09-15T20:31:31.118 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-09-15T20:31:31.114+0000 7f197e0257c0 -1 Falling back to public interface 2024-09-15T20:31:33.697 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-09-15T20:31:33.694+0000 7f4f7b060640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-09-15T20:31:33.704 INFO:tasks.ceph.mgr.x.smithi146.stderr:2024-09-15T20:31:33.698+0000 7f4f7b060640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-09-15T20:31:34.575 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-15T20:31:34.570+0000 7fae8c4ff7c0 -1 osd.3 0 log_to_monitors true 2024-09-15T20:31:34.616 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-09-15T20:31:34.612+0000 7ff6a45487c0 -1 osd.1 0 log_to_monitors true 2024-09-15T20:31:34.784 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-09-15T20:31:34.780+0000 7ff7a09567c0 -1 osd.0 0 log_to_monitors true 2024-09-15T20:31:34.907 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-09-15T20:31:34.902+0000 7fbcd83537c0 -1 osd.5 0 log_to_monitors true 2024-09-15T20:31:35.065 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T20:31:35.060+0000 7fc120f337c0 -1 osd.2 0 log_to_monitors true 2024-09-15T20:31:35.114 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-09-15T20:31:35.110+0000 7f197e0257c0 -1 osd.4 0 log_to_monitors true 2024-09-15T20:31:35.720 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-15T20:31:35.714+0000 7fae884a8640 -1 osd.3 0 waiting for initial osdmap 2024-09-15T20:31:35.735 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-15T20:31:35.730+0000 7fae842e2640 -1 osd.3 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:31:36.379 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-09-15T20:31:36.376+0000 7ff6a0d03640 -1 osd.1 0 waiting for initial osdmap 2024-09-15T20:31:36.385 INFO:tasks.ceph.osd.1.smithi078.stderr:2024-09-15T20:31:36.384+0000 7ff698323640 -1 osd.1 10 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:31:36.716 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T20:31:36.712+0000 7fc11cedc640 -1 osd.2 0 waiting for initial osdmap 2024-09-15T20:31:36.729 INFO:tasks.ceph.osd.2.smithi078.stderr:2024-09-15T20:31:36.728+0000 7fc118d16640 -1 osd.2 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:31:37.098 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T20:31:37.224 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.220+0000 7fc7e16bd640 10 monclient: get_monmap_and_config 2024-09-15T20:31:37.224 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.220+0000 7fc7e16bd640 10 monclient: build_initial_monmap 2024-09-15T20:31:37.224 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.220+0000 7fc7e16bd640 10 monclient: monmap: 2024-09-15T20:31:37.224 INFO:teuthology.misc.health.smithi078.stderr:epoch 0 2024-09-15T20:31:37.224 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:37.225 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:37.223731+0000 2024-09-15T20:31:37.225 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:37.223731+0000 2024-09-15T20:31:37.225 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:37.225 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:37.225 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:37.225 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:37.225 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:37.225 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:37.228 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.220+0000 7fc7e16bd640 1 Processor -- start 2024-09-15T20:31:37.228 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.220+0000 7fc7e16bd640 1 -- start start 2024-09-15T20:31:37.228 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.220+0000 7fc7e16bd640 10 monclient: init 2024-09-15T20:31:37.228 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.220+0000 7fc7e16bd640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:37.228 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.220+0000 7fc7e16bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:37.228 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc136c90 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc137070 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc137500 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc0a4de0 con 0x7fc7dc137070 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc0a50c0 con 0x7fc7dc136c90 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc0a53a0 con 0x7fc7dc137500 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient(hunting): authenticate will time out at 1158.108127s 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7da7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc7dc137070 0x7fc7dc13cce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44584/0 (socket says 172.21.15.78:44584) 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7da7fc640 1 -- 172.21.15.78:0/45571830 learned_addr learned my addr 172.21.15.78:0/45571830 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1941830842 0 0) 0x7fc7dc0a53a0 con 0x7fc7dc137500 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:37.229 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): my global_id is 4208 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7c4003530 con 0x7fc7dc137500 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3545857285 0 0) 0x7fc7dc0a4de0 con 0x7fc7dc137070 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): my global_id is 4203 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc0a53a0 con 0x7fc7dc137070 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3395689424 0 0) 0x7fc7dc0a50c0 con 0x7fc7dc136c90 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:37.230 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient(hunting): my global_id is 4210 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc0a4de0 con 0x7fc7dc136c90 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3061525407 0 0) 0x7fc7dc0a53a0 con 0x7fc7dc137070 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 >> v1:172.21.15.78:6790/0 conn(0x7fc7dc137500 legacy=0x7fc7dc140410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 >> v1:172.21.15.146:6789/0 conn(0x7fc7dc136c90 legacy=0x7fc7dc0a6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 monclient: found mon.noname-a 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc09d8d0 con 0x7fc7dc137070 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:37.231 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc7d0001600 con 0x7fc7dc137070 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: dump: 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:37.232 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 5 monclient: authenticate success, global_id 4203 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7d0001820 con 0x7fc7dc137070 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 1 -- 172.21.15.78:0/45571830 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc7d0001c30 con 0x7fc7dc137070 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 4 set_mon_vals no callback set 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7d9ffb640 10 monclient: dump: 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:37.233 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient: shutdown 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- 172.21.15.78:0/45571830 >> v1:172.21.15.78:6789/0 conn(0x7fc7dc137070 legacy=0x7fc7dc13cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- 172.21.15.78:0/45571830 shutdown_connections 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- 172.21.15.78:0/45571830 >> 172.21.15.78:0/45571830 conn(0x7fc7dc0aa360 msgr2=0x7fc7dc0aa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- 172.21.15.78:0/45571830 shutdown_connections 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- 172.21.15.78:0/45571830 wait complete. 2024-09-15T20:31:37.234 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient: build_initial_monmap 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient: monmap: 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:epoch 0 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:created 0.000000 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:37.235 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 librados: starting msgr at 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 librados: starting objecter 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 Processor -- start 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 -- start start 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 librados: setting wanted keys 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 1 librados: calling monclient init 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.224+0000 7fc7e16bd640 10 monclient: init 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:37.239 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient(hunting): picked mon.noname-b con 0x7fc7dc136c90 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient(hunting): picked mon.noname-a con 0x7fc7dc137070 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient(hunting): picked mon.noname-c con 0x7fc7dc137500 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7da7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc7dc137070 0x7fc7dc1319b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44588/0 (socket says 172.21.15.78:44588) 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc133140 con 0x7fc7dc137070 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7da7fc640 1 -- 172.21.15.78:0/3295446723 learned_addr learned my addr 172.21.15.78:0/3295446723 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 client.?.objecter ms_handle_connect 0x7fc7dc137070 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 client.?.objecter ms_handle_connect 0x7fc7dc137500 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc06e6d0 con 0x7fc7dc136c90 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc7dc134a70 con 0x7fc7dc137500 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 client.?.objecter ms_handle_connect 0x7fc7dc136c90 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:37.240 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient(hunting): authenticate will time out at 1158.112897s 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877244969 0 0) 0x7fc7dc133140 con 0x7fc7dc137070 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): my global_id is 4206 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7c8004ec0 con 0x7fc7dc137070 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 121459331 0 0) 0x7fc7dc134a70 con 0x7fc7dc137500 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): my global_id is 4211 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc133140 con 0x7fc7dc137500 2024-09-15T20:31:37.241 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 766724967 0 0) 0x7fc7dc0a4de0 con 0x7fc7dc136c90 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): my global_id is 4213 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc7dc134a70 con 0x7fc7dc136c90 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4278346266 0 0) 0x7fc7c8004ec0 con 0x7fc7dc137070 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7dc0a4de0 con 0x7fc7dc137070 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc7d0001c30 con 0x7fc7dc137070 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): dump: 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:37.242 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3488837996 0 0) 0x7fc7dc133140 con 0x7fc7dc137500 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc7c8004ec0 con 0x7fc7dc137500 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc7d40014e0 con 0x7fc7dc137500 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient(hunting): dump: 2024-09-15T20:31:37.243 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 329008625 0 0) 0x7fc7dc0a4de0 con 0x7fc7dc137070 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 >> v1:172.21.15.78:6790/0 conn(0x7fc7dc137500 legacy=0x7fc7dc1d4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 >> v1:172.21.15.146:6789/0 conn(0x7fc7dc136c90 legacy=0x7fc7dc1312a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:37.244 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 monclient: found mon.a 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7dc134d50 con 0x7fc7dc137070 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 5 monclient: authenticate success, global_id 4206 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient: _renew_subs 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc7dc132a10 con 0x7fc7dc137070 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 client.4206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient: _renew_subs 2024-09-15T20:31:37.245 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc7dc134790 con 0x7fc7dc137070 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 1 librados: init done 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc7d0001780 con 0x7fc7dc137070 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc7d0004da0 con 0x7fc7dc137070 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 monclient: dump: 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:37.246 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7d97fa640 4 set_mon_vals no callback set 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7bc005090 con 0x7fc7dc137070 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 4) ==== 48776+0+0 (unknown 1094589517 0 0) 0x7fc7d0010dd0 con 0x7fc7dc137070 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(11..11 src has 1..11) ==== 2207+0+0 (unknown 1002926350 0 0) 0x7fc7d004aac0 con 0x7fc7dc137070 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 client.4206.objecter ms_dispatch 0x7fc7dc000ba0 osd_map(11..11 src has 1..11) 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 3 client.4206.objecter handle_osd_map got epochs [11,11] > 0 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 3 client.4206.objecter handle_osd_map decoding full epoch 11 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 20 client.4206.objecter dump_active .. 0 homeless 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7daffd640 10 monclient: get_auth_request con 0x7fc7c803e3b0 auth_method 0 2024-09-15T20:31:37.247 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.228+0000 7fc7c37fe640 10 client.4206.objecter ms_handle_connect 0x7fc7c803e3b0 2024-09-15T20:31:37.248 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.236+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc7d0016b50 con 0x7fc7dc137070 2024-09-15T20:31:37.248 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.236+0000 7fc7c37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:37.248 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.236+0000 7fc7c37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:37.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:37.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:37.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:37.418 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc7bc005380 con 0x7fc7dc137070 2024-09-15T20:31:37.419 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7c37fe640 1 -- 172.21.15.78:0/3295446723 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) ==== 74+0+6097 (unknown 238102980 0 1078941844) 0x7fc7d0016450 con 0x7fc7dc137070 2024-09-15T20:31:37.419 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7c37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:37.419 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7c37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:37.419 INFO:teuthology.misc.health.smithi078.stdout: 2024-09-15T20:31:37.419 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":11,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","created":"2024-09-15T20:31:27.702188+0000","modified":"2024-09-15T20:31:36.702320+0000","last_up_change":"2024-09-15T20:31:36.702320+0000","last_in_change":"2024-09-15T20:31:29.437051+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9003b16b-6a87-4659-8436-ff541ea8dfcb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"be602c68-218b-4573-8d19-eef568af9d4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6800","nonce":4006351464}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6801","nonce":4006351464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6803","nonce":4006351464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6802","nonce":4006351464}]},"public_addr":"172.21.15.78:6800/4006351464","cluster_addr":"172.21.15.78:6801/4006351464","heartbeat_back_addr":"172.21.15.78:6803/4006351464","heartbeat_front_addr":"172.21.15.78:6802/4006351464","state":["exists","up"]},{"osd":2,"uuid":"1fab92e2-4880-4f66-86f9-1c59752850a1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"74c2a9ff-0635-4664-9dc5-d156f4b60bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6800","nonce":490126554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6801","nonce":490126554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6803","nonce":490126554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6802","nonce":490126554}]},"public_addr":"172.21.15.146:6800/490126554","cluster_addr":"172.21.15.146:6801/490126554","heartbeat_back_addr":"172.21.15.146:6803/490126554","heartbeat_front_addr":"172.21.15.146:6802/490126554","state":["exists","up"]},{"osd":4,"uuid":"0abc805d-20bc-40e6-a25a-0d9694ace9eb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"08c34b4f-deb1-401a-aab7-00854ae6712f","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:31:37.421 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 10 librados: watch_flush enter 2024-09-15T20:31:37.421 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 10 librados: watch_flush exit 2024-09-15T20:31:37.421 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 20 client.4206.objecter shutdown clearing up homeless session... 2024-09-15T20:31:37.421 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 10 client.4206.objecter successfully canceled tick 2024-09-15T20:31:37.421 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 >> v1:172.21.15.146:6812/435598642 conn(0x7fc7c803e3b0 legacy=0x7fc7c8040870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:37.421 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 10 monclient: shutdown 2024-09-15T20:31:37.421 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:37.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 >> v1:172.21.15.78:6789/0 conn(0x7fc7dc137070 legacy=0x7fc7dc1319b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:37.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 shutdown_connections 2024-09-15T20:31:37.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 >> 172.21.15.78:0/3295446723 conn(0x7fc7dc0aa360 msgr2=0x7fc7dc12fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:37.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 shutdown_connections 2024-09-15T20:31:37.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 1 -- 172.21.15.78:0/3295446723 wait complete. 2024-09-15T20:31:37.422 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:37.416+0000 7fc7e16bd640 1 librados: shutdown 2024-09-15T20:31:37.429 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-09-15T20:31:37.428+0000 7ff79c8ff640 -1 osd.0 0 waiting for initial osdmap 2024-09-15T20:31:37.433 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-09-15T20:31:37.428+0000 7ff794731640 -1 osd.0 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:31:37.437 DEBUG:teuthology.misc:2 of 6 OSDs are up 2024-09-15T20:31:37.553 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-09-15T20:31:37.550+0000 7f1979fce640 -1 osd.4 0 waiting for initial osdmap 2024-09-15T20:31:37.556 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-09-15T20:31:37.550+0000 7f1971e00640 -1 osd.4 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:31:37.591 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-09-15T20:31:37.586+0000 7fbcd4b0e640 -1 osd.5 0 waiting for initial osdmap 2024-09-15T20:31:37.596 INFO:tasks.ceph.osd.5.smithi146.stderr:2024-09-15T20:31:37.590+0000 7fbccc12e640 -1 osd.5 11 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-15T20:31:44.042 DEBUG:teuthology.orchestra.run.smithi078:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-09-15T20:31:44.164 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.160+0000 7f0df140a640 10 monclient: get_monmap_and_config 2024-09-15T20:31:44.164 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.160+0000 7f0df140a640 10 monclient: build_initial_monmap 2024-09-15T20:31:44.164 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.160+0000 7f0df140a640 10 monclient: monmap: 2024-09-15T20:31:44.164 INFO:teuthology.misc.health.smithi078.stderr:epoch 0 2024-09-15T20:31:44.164 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.164 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:44.163597+0000 2024-09-15T20:31:44.165 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:44.163597+0000 2024-09-15T20:31:44.165 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:44.165 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.165 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:44.165 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:44.165 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:44.165 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:44.168 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.160+0000 7f0df140a640 1 Processor -- start 2024-09-15T20:31:44.168 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.160+0000 7f0df140a640 1 -- start start 2024-09-15T20:31:44.168 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.160+0000 7f0df140a640 10 monclient: init 2024-09-15T20:31:44.168 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): picked mon.noname-c con 0x7f0dec0a1910 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): picked mon.noname-b con 0x7f0dec09df80 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): picked mon.noname-a con 0x7f0dec0a2400 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec09dd40 con 0x7f0dec0a2400 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec09dac0 con 0x7f0dec0a1910 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec09ede0 con 0x7f0dec09df80 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): authenticate will time out at 1165.048520s 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0dea7fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f0dec09df80 0x7f0dec13aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39098/0 (socket says 172.21.15.78:39098) 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0dea7fc640 1 -- 172.21.15.78:0/2833962972 learned_addr learned my addr 172.21.15.78:0/2833962972 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:44.169 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3162024638 0 0) 0x7f0dec09ede0 con 0x7f0dec09df80 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient(hunting): my global_id is 4223 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dd8003530 con 0x7f0dec09df80 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729298668 0 0) 0x7f0dec09dd40 con 0x7f0dec0a2400 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient(hunting): my global_id is 4218 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec09ede0 con 0x7f0dec0a2400 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3112211249 0 0) 0x7f0dd8003530 con 0x7f0dec09df80 2024-09-15T20:31:44.170 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 >> v1:172.21.15.146:6789/0 conn(0x7f0dec0a1910 legacy=0x7f0dec0a1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 >> v1:172.21.15.78:6789/0 conn(0x7f0dec0a2400 legacy=0x7f0dec13e2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 monclient: found mon.noname-b 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec0a33d0 con 0x7f0dec09df80 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 5 monclient: authenticate success, global_id 4223 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0dd40016a0 con 0x7f0dec09df80 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:44.171 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: dump: 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:44.172 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0dd4001840 con 0x7f0dec09df80 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 1 -- 172.21.15.78:0/2833962972 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0dd4001cd0 con 0x7f0dec09df80 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de9ffb640 10 monclient: dump: 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.173 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 4 set_mon_vals no callback set 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: shutdown 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- 172.21.15.78:0/2833962972 >> v1:172.21.15.78:6790/0 conn(0x7f0dec09df80 legacy=0x7f0dec13aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- 172.21.15.78:0/2833962972 shutdown_connections 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- 172.21.15.78:0/2833962972 >> 172.21.15.78:0/2833962972 conn(0x7f0dec0a6500 msgr2=0x7f0dec0a2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- 172.21.15.78:0/2833962972 shutdown_connections 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- 172.21.15.78:0/2833962972 wait complete. 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: build_initial_monmap 2024-09-15T20:31:44.174 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: monmap: 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:epoch 0 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:created 0.000000 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 librados: starting msgr at 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 librados: starting objecter 2024-09-15T20:31:44.175 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 Processor -- start 2024-09-15T20:31:44.176 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- start start 2024-09-15T20:31:44.176 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 librados: setting wanted keys 2024-09-15T20:31:44.176 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 librados: calling monclient init 2024-09-15T20:31:44.176 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: init 2024-09-15T20:31:44.179 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:44.179 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): picked mon.noname-c con 0x7f0dec09df80 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): picked mon.noname-a con 0x7f0dec0a1910 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): picked mon.noname-b con 0x7f0dec0a2400 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec11d6e0 con 0x7f0dec0a1910 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec136770 con 0x7f0dec0a2400 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0dec0705f0 con 0x7f0dec09df80 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0df140a640 10 monclient(hunting): authenticate will time out at 1165.051653s 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0dea7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0dec0a1910 0x7f0dec1cf620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37874/0 (socket says 172.21.15.78:37874) 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0deaffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f0dec09df80 0x7f0dec136060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39110/0 (socket says 172.21.15.78:39110) 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0dea7fc640 1 -- 172.21.15.78:0/3125203328 learned_addr learned my addr 172.21.15.78:0/3125203328 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:44.180 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de37fe640 10 client.?.objecter ms_handle_connect 0x7f0dec0a1910 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de37fe640 10 client.?.objecter ms_handle_connect 0x7f0dec09df80 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.164+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2741339137 0 0) 0x7f0dec11d6e0 con 0x7f0dec0a1910 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): my global_id is 4221 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dcc004ec0 con 0x7f0dec0a1910 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808687986 0 0) 0x7f0dec0705f0 con 0x7f0dec09df80 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.181 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): my global_id is 4226 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec11d6e0 con 0x7f0dec09df80 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 client.?.objecter ms_handle_connect 0x7f0dec0a2400 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4269812892 0 0) 0x7f0dcc004ec0 con 0x7f0dec0a1910 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dec0705f0 con 0x7f0dec0a1910 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0dd4001bd0 con 0x7f0dec0a1910 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): dump: 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.182 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1376770382 0 0) 0x7f0dec11d6e0 con 0x7f0dec09df80 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0dcc004ec0 con 0x7f0dec09df80 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0ddc001cc0 con 0x7f0dec09df80 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): dump: 2024-09-15T20:31:44.183 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1905232405 0 0) 0x7f0dec136770 con 0x7f0dec0a2400 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.184 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient(hunting): my global_id is 4228 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0dec11d6e0 con 0x7f0dec0a2400 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2257276820 0 0) 0x7f0dec0705f0 con 0x7f0dec0a1910 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 >> v1:172.21.15.78:6790/0 conn(0x7f0dec09df80 legacy=0x7f0dec136060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 >> v1:172.21.15.146:6789/0 conn(0x7f0dec0a2400 legacy=0x7f0dec1d2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 monclient: found mon.a 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dec09d0f0 con 0x7f0dec0a1910 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:44.185 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 5 monclient: authenticate success, global_id 4221 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 10 monclient: _renew_subs 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0dec11c5e0 con 0x7f0dec0a1910 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 10 client.4221.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 10 monclient: _renew_subs 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0dec13ea10 con 0x7f0dec0a1910 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 1 librados: init done 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0dd4001820 con 0x7f0dec0a1910 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0dd4004da0 con 0x7f0dec0a1910 2024-09-15T20:31:44.186 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de97fa640 4 set_mon_vals no callback set 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 monclient: dump: 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:epoch 1 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr: 2024-09-15T20:31:44.187 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd0005090 con 0x7f0dec0a1910 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f0dd4010e10 con 0x7f0dec0a1910 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 1962966789 0 0) 0x7f0dd404aec0 con 0x7f0dec0a1910 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 client.4221.objecter ms_dispatch 0x7f0dec000ba0 osd_map(15..15 src has 1..15) 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 3 client.4221.objecter handle_osd_map got epochs [15,15] > 0 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 3 client.4221.objecter handle_osd_map decoding full epoch 15 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 20 client.4221.objecter dump_active .. 0 homeless 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0deaffd640 10 monclient: get_auth_request con 0x7f0dcc03e290 auth_method 0 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.168+0000 7f0de37fe640 10 client.4221.objecter ms_handle_connect 0x7f0dcc03e290 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.172+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f0dd4016b50 con 0x7f0dec0a1910 2024-09-15T20:31:44.188 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.172+0000 7f0de37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:44.189 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.172+0000 7f0de37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:44.340 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.336+0000 7f0df140a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:44.340 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.336+0000 7f0df140a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:44.340 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.336+0000 7f0df140a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.341 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.336+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0dd0005380 con 0x7f0dec0a1910 2024-09-15T20:31:44.341 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.336+0000 7f0de37fe640 1 -- 172.21.15.78:0/3125203328 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) ==== 74+0+9033 (unknown 4221056075 0 2769398291) 0x7f0dd4016450 con 0x7f0dec0a1910 2024-09-15T20:31:44.341 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.336+0000 7f0de37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:44.341 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.336+0000 7f0de37fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:44.341 INFO:teuthology.misc.health.smithi078.stdout: 2024-09-15T20:31:44.342 INFO:teuthology.misc.health.smithi078.stdout:{"epoch":15,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","created":"2024-09-15T20:31:27.702188+0000","modified":"2024-09-15T20:31:40.744601+0000","last_up_change":"2024-09-15T20:31:37.715594+0000","last_in_change":"2024-09-15T20:31:29.437051+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:31:38.725860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9003b16b-6a87-4659-8436-ff541ea8dfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6804","nonce":3648339548}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6805","nonce":3648339548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6807","nonce":3648339548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6806","nonce":3648339548}]},"public_addr":"172.21.15.78:6804/3648339548","cluster_addr":"172.21.15.78:6805/3648339548","heartbeat_back_addr":"172.21.15.78:6807/3648339548","heartbeat_front_addr":"172.21.15.78:6806/3648339548","state":["exists","up"]},{"osd":1,"uuid":"be602c68-218b-4573-8d19-eef568af9d4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6800","nonce":4006351464}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6801","nonce":4006351464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6803","nonce":4006351464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6802","nonce":4006351464}]},"public_addr":"172.21.15.78:6800/4006351464","cluster_addr":"172.21.15.78:6801/4006351464","heartbeat_back_addr":"172.21.15.78:6803/4006351464","heartbeat_front_addr":"172.21.15.78:6802/4006351464","state":["exists","up"]},{"osd":2,"uuid":"1fab92e2-4880-4f66-86f9-1c59752850a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6808","nonce":404661851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6809","nonce":404661851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6811","nonce":404661851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6810","nonce":404661851}]},"public_addr":"172.21.15.78:6808/404661851","cluster_addr":"172.21.15.78:6809/404661851","heartbeat_back_addr":"172.21.15.78:6811/404661851","heartbeat_front_addr":"172.21.15.78:6810/404661851","state":["exists","up"]},{"osd":3,"uuid":"74c2a9ff-0635-4664-9dc5-d156f4b60bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6800","nonce":490126554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6801","nonce":490126554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6803","nonce":490126554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6802","nonce":490126554}]},"public_addr":"172.21.15.146:6800/490126554","cluster_addr":"172.21.15.146:6801/490126554","heartbeat_back_addr":"172.21.15.146:6803/490126554","heartbeat_front_addr":"172.21.15.146:6802/490126554","state":["exists","up"]},{"osd":4,"uuid":"0abc805d-20bc-40e6-a25a-0d9694ace9eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6808","nonce":2960966471}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6809","nonce":2960966471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6811","nonce":2960966471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6810","nonce":2960966471}]},"public_addr":"172.21.15.146:6808/2960966471","cluster_addr":"172.21.15.146:6809/2960966471","heartbeat_back_addr":"172.21.15.146:6811/2960966471","heartbeat_front_addr":"172.21.15.146:6810/2960966471","state":["exists","up"]},{"osd":5,"uuid":"08c34b4f-deb1-401a-aab7-00854ae6712f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6804","nonce":516207226}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6805","nonce":516207226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6807","nonce":516207226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6806","nonce":516207226}]},"public_addr":"172.21.15.146:6804/516207226","cluster_addr":"172.21.15.146:6805/516207226","heartbeat_back_addr":"172.21.15.146:6807/516207226","heartbeat_front_addr":"172.21.15.146:6806/516207226","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.779852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.576609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.081865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.582230+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.088303+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.878442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:31:44.343 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 10 librados: watch_flush enter 2024-09-15T20:31:44.343 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 10 librados: watch_flush exit 2024-09-15T20:31:44.343 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 20 client.4221.objecter shutdown clearing up homeless session... 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 10 client.4221.objecter successfully canceled tick 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 >> v1:172.21.15.146:6812/435598642 conn(0x7f0dcc03e290 legacy=0x7f0dcc040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 10 monclient: shutdown 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 >> v1:172.21.15.78:6789/0 conn(0x7f0dec0a1910 legacy=0x7f0dec1cf620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 shutdown_connections 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 >> 172.21.15.78:0/3125203328 conn(0x7f0dec0a6500 msgr2=0x7f0dec0a4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 shutdown_connections 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 1 -- 172.21.15.78:0/3125203328 wait complete. 2024-09-15T20:31:44.344 INFO:teuthology.misc.health.smithi078.stderr:2024-09-15T20:31:44.340+0000 7f0df140a640 1 librados: shutdown 2024-09-15T20:31:44.357 DEBUG:teuthology.misc:6 of 6 OSDs are up 2024-09-15T20:31:44.357 INFO:tasks.ceph:Creating RBD pool 2024-09-15T20:31:44.357 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph osd pool create rbd 8 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.472+0000 7fb078ed5640 10 monclient: get_monmap_and_config 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.472+0000 7fb078ed5640 10 monclient: build_initial_monmap 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.472+0000 7fb078ed5640 10 monclient: monmap: 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:44.476353+0000 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:44.476353+0000 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.477 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:44.478 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:44.478 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:44.478 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 Processor -- start 2024-09-15T20:31:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- start start 2024-09-15T20:31:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient: init 2024-09-15T20:31:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient(hunting): picked mon.noname-c con 0x7fb074135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:44.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient(hunting): picked mon.noname-a con 0x7fb0740a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient(hunting): picked mon.noname-b con 0x7fb07413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb0740745e0 con 0x7fb0740a24e0 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07406ecf0 con 0x7fb074135b60 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07406efd0 con 0x7fb07413c220 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient(hunting): authenticate will time out at 1165.360874s 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb072d9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb07413c220 0x7fb07413e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39120/0 (socket says 172.21.15.78:39120) 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb072d9d640 1 -- 172.21.15.78:0/3955500621 learned_addr learned my addr 172.21.15.78:0/3955500621 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb06bfff640 1 --1- 172.21.15.78:0/3955500621 >> v1:172.21.15.78:6789/0 conn(0x7fb0740a24e0 0x7fb07413ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37876/0 (socket says 172.21.15.78:37876) 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 326604539 0 0) 0x7fb07406efd0 con 0x7fb07413c220 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient(hunting): my global_id is 4229 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb060003530 con 0x7fb07413c220 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1273142047 0 0) 0x7fb0740745e0 con 0x7fb0740a24e0 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient(hunting): my global_id is 4224 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb07406efd0 con 0x7fb0740a24e0 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1621520037 0 0) 0x7fb060003530 con 0x7fb07413c220 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 >> v1:172.21.15.146:6789/0 conn(0x7fb074135b60 legacy=0x7fb074135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 >> v1:172.21.15.78:6789/0 conn(0x7fb0740a24e0 legacy=0x7fb07413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: _finish_hunting 0 2024-09-15T20:31:44.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 monclient: found mon.noname-b 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb074136650 con 0x7fb07413c220 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: _finish_auth 0 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: _check_auth_tickets 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb06c001a20 con 0x7fb07413c220 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: dump: 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.483 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 5 monclient: authenticate success, global_id 4229 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb06c001bc0 con 0x7fb07413c220 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:44.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 1 -- 172.21.15.78:0/3955500621 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb06c001f90 con 0x7fb07413c220 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb071d9b640 10 monclient: dump: 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.485 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 4 set_mon_vals no callback set 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient: shutdown 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- 172.21.15.78:0/3955500621 >> v1:172.21.15.78:6790/0 conn(0x7fb07413c220 legacy=0x7fb07413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- 172.21.15.78:0/3955500621 shutdown_connections 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- 172.21.15.78:0/3955500621 >> 172.21.15.78:0/3955500621 conn(0x7fb0740a63c0 msgr2=0x7fb0740a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- 172.21.15.78:0/3955500621 shutdown_connections 2024-09-15T20:31:44.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- 172.21.15.78:0/3955500621 wait complete. 2024-09-15T20:31:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient: build_initial_monmap 2024-09-15T20:31:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient: monmap: 2024-09-15T20:31:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:44.490 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 librados: starting msgr at 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 librados: starting objecter 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 Processor -- start 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 -- start start 2024-09-15T20:31:44.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 librados: setting wanted keys 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 1 librados: calling monclient init 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.476+0000 7fb078ed5640 10 monclient: init 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient(hunting): picked mon.noname-b con 0x7fb0740a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient(hunting): picked mon.noname-c con 0x7fb074135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient(hunting): picked mon.noname-a con 0x7fb07413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb074073860 con 0x7fb07413c220 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07409d180 con 0x7fb0740a24e0 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb07413b8f0 con 0x7fb074135b60 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient(hunting): authenticate will time out at 1165.364200s 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06bfff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb074135b60 0x7fb074072710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39134/0 (socket says 172.21.15.78:39134) 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06bfff640 1 -- 172.21.15.78:0/1228038907 learned_addr learned my addr 172.21.15.78:0/1228038907 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:44.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 client.?.objecter ms_handle_connect 0x7fb074135b60 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 client.?.objecter ms_handle_connect 0x7fb07413c220 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407281020 0 0) 0x7fb07413b8f0 con 0x7fb074135b60 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): my global_id is 4232 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb054004ec0 con 0x7fb074135b60 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2865368857 0 0) 0x7fb074073860 con 0x7fb07413c220 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): my global_id is 4227 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb07413b8f0 con 0x7fb07413c220 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 client.?.objecter ms_handle_connect 0x7fb0740a24e0 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2004861069 0 0) 0x7fb07413b8f0 con 0x7fb07413c220 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb074073860 con 0x7fb07413c220 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb06c0018e0 con 0x7fb07413c220 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): dump: 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.494 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 360728103 0 0) 0x7fb054004ec0 con 0x7fb074135b60 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb07413b8f0 con 0x7fb074135b60 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232823975 0 0) 0x7fb07406ecf0 con 0x7fb0740a24e0 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): my global_id is 4234 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb054004ec0 con 0x7fb0740a24e0 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb05c001940 con 0x7fb074135b60 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient(hunting): dump: 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.496 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3035655846 0 0) 0x7fb074073860 con 0x7fb07413c220 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 >> v1:172.21.15.78:6790/0 conn(0x7fb074135b60 legacy=0x7fb074072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 >> v1:172.21.15.146:6789/0 conn(0x7fb0740a24e0 legacy=0x7fb074072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 monclient: found mon.a 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb074137dd0 con 0x7fb07413c220 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient: _finish_auth 0 2024-09-15T20:31:44.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 5 monclient: authenticate success, global_id 4227 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient: _renew_subs 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb0741380b0 con 0x7fb07413c220 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 client.4227.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient: _renew_subs 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb074072e20 con 0x7fb07413c220 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 1 librados: init done 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb06c001b40 con 0x7fb07413c220 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:44.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb06c004e20 con 0x7fb07413c220 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb07159a640 4 set_mon_vals no callback set 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 monclient: dump: 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.499 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb058005090 con 0x7fb07413c220 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb06c010e90 con 0x7fb07413c220 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(15..15 src has 1..15) ==== 3153+0+0 (unknown 1962966789 0 0) 0x7fb06c04b100 con 0x7fb07413c220 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 client.4227.objecter ms_dispatch 0x7fb074000ba0 osd_map(15..15 src has 1..15) 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 3 client.4227.objecter handle_osd_map got epochs [15,15] > 0 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 3 client.4227.objecter handle_osd_map decoding full epoch 15 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 20 client.4227.objecter dump_active .. 0 homeless 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb07259c640 10 monclient: get_auth_request con 0x7fb05403e240 auth_method 0 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.480+0000 7fb06affd640 10 client.4227.objecter ms_handle_connect 0x7fb05403e240 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.484+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb06c016c70 con 0x7fb07413c220 2024-09-15T20:31:44.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.484+0000 7fb06affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:44.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.484+0000 7fb06affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:44.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.644+0000 7fb078ed5640 10 start_mon_command cmd=[{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T20:31:44.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.644+0000 7fb078ed5640 10 monclient: _send_command 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T20:31:44.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.644+0000 7fb078ed5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.644+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) -- 0x7fb058005c50 con 0x7fb07413c220 2024-09-15T20:31:44.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.780+0000 7fb06affd640 1 -- 172.21.15.78:0/1228038907 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) ==== 109+0+0 (unknown 231814880 0 0) 0x7fb06c01aff0 con 0x7fb07413c220 2024-09-15T20:31:44.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.780+0000 7fb06affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}] 2024-09-15T20:31:44.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.780+0000 7fb06affd640 10 monclient: _finish_command 2 = system:0 pool 'rbd' created 2024-09-15T20:31:44.782 INFO:teuthology.orchestra.run.smithi078.stderr:pool 'rbd' created 2024-09-15T20:31:44.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 10 librados: watch_flush enter 2024-09-15T20:31:44.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 10 librados: watch_flush exit 2024-09-15T20:31:44.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 20 client.4227.objecter shutdown clearing up homeless session... 2024-09-15T20:31:44.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 10 client.4227.objecter successfully canceled tick 2024-09-15T20:31:44.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 >> v1:172.21.15.146:6812/435598642 conn(0x7fb05403e240 legacy=0x7fb054040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 10 monclient: shutdown 2024-09-15T20:31:44.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:44.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 >> v1:172.21.15.78:6789/0 conn(0x7fb07413c220 legacy=0x7fb0741d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 shutdown_connections 2024-09-15T20:31:44.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 >> 172.21.15.78:0/1228038907 conn(0x7fb0740a63c0 msgr2=0x7fb0740a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:44.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 shutdown_connections 2024-09-15T20:31:44.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 1 -- 172.21.15.78:0/1228038907 wait complete. 2024-09-15T20:31:44.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.784+0000 7fb078ed5640 1 librados: shutdown 2024-09-15T20:31:44.804 DEBUG:teuthology.orchestra.run.smithi078:> rbd --cluster ceph pool init rbd 2024-09-15T20:31:44.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.856+0000 7f042eb26100 10 monclient: get_monmap_and_config 2024-09-15T20:31:44.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.856+0000 7f042eb26100 10 monclient: build_initial_monmap 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.856+0000 7f042eb26100 10 monclient: monmap: 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:44.860018+0000 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:44.860018+0000 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:44.861 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:44.862 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 Processor -- start 2024-09-15T20:31:44.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- start start 2024-09-15T20:31:44.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient: init 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient: _reopen_session rank -1 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient(hunting): picked mon.noname-c con 0x55965f6beab0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient(hunting): picked mon.noname-b con 0x55965f62d160 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient(hunting): picked mon.noname-a con 0x55965f629350 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55965f476c80 con 0x55965f629350 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55965f6bf5a0 con 0x55965f6beab0 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55965f62cb70 con 0x55965f62d160 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient(hunting): _renew_subs 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient(hunting): authenticate will time out at 1165.744987s 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042ce2e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x55965f62d160 0x55965f6c3a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39136/0 (socket says 172.21.15.78:39136) 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042e891640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x55965f629350 0x55965f6c7140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37892/0 (socket says 172.21.15.78:37892) 2024-09-15T20:31:44.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042ce2e640 1 -- 172.21.15.78:0/1725577720 learned_addr learned my addr 172.21.15.78:0/1725577720 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3023541777 0 0) 0x55965f62cb70 con 0x55965f62d160 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient(hunting): my global_id is 4235 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0414001e20 con 0x55965f62d160 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3119059936 0 0) 0x55965f476c80 con 0x55965f629350 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient(hunting): my global_id is 4230 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55965f62cb70 con 0x55965f629350 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3903857649 0 0) 0x55965f62cb70 con 0x55965f629350 2024-09-15T20:31:44.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 >> v1:172.21.15.78:6790/0 conn(0x55965f62d160 legacy=0x55965f6c3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 >> v1:172.21.15.146:6789/0 conn(0x55965f6beab0 legacy=0x55965f6bee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 monclient: found mon.noname-a 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55965f6107d0 con 0x55965f629350 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: _finish_auth 0 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0420001980 con 0x55965f629350 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 5 monclient: authenticate success, global_id 4230 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:44.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: dump: 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:44.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0420001b20 con 0x55965f629350 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 1 -- 172.21.15.78:0/1725577720 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0420001fb0 con 0x55965f629350 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f041ffff640 10 monclient: dump: 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.869 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient: get_monmap_and_config success 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 4 set_mon_vals no callback set 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient: shutdown 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- 172.21.15.78:0/1725577720 >> v1:172.21.15.78:6789/0 conn(0x55965f629350 legacy=0x55965f6c7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- 172.21.15.78:0/1725577720 shutdown_connections 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- 172.21.15.78:0/1725577720 >> 172.21.15.78:0/1725577720 conn(0x55965f60f900 msgr2=0x55965f60fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- 172.21.15.78:0/1725577720 shutdown_connections 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- 172.21.15.78:0/1725577720 wait complete. 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient: build_initial_monmap 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient: monmap: 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:44.870 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:44.871 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:44.871 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:44.871 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:44.871 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.871 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:44.871 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:44.871 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:44.871 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 librados: starting msgr at 2024-09-15T20:31:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 librados: starting objecter 2024-09-15T20:31:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 Processor -- start 2024-09-15T20:31:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 -- start start 2024-09-15T20:31:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 librados: setting wanted keys 2024-09-15T20:31:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 1 librados: calling monclient init 2024-09-15T20:31:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.860+0000 7f042eb26100 10 monclient: init 2024-09-15T20:31:44.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient: _reopen_session rank -1 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient(hunting): picked mon.noname-c con 0x55965f629350 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient(hunting): picked mon.noname-b con 0x55965f62d160 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient(hunting): picked mon.noname-a con 0x55965f6beab0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55965f60f420 con 0x55965f6beab0 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55965f6bdf50 con 0x55965f62d160 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x55965f6259d0 con 0x55965f629350 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient(hunting): _renew_subs 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient(hunting): authenticate will time out at 1165.748498s 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042d62f640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x55965f629350 0x55965f6bd130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39148/0 (socket says 172.21.15.78:39148) 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042d62f640 1 -- 172.21.15.78:0/3300934940 learned_addr learned my addr 172.21.15.78:0/3300934940 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 client.?.objecter ms_handle_connect 0x55965f629350 2024-09-15T20:31:44.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 client.?.objecter ms_handle_connect 0x55965f6beab0 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4069620338 0 0) 0x55965f6259d0 con 0x55965f629350 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): my global_id is 4238 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f040c002ff0 con 0x55965f629350 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3601438488 0 0) 0x55965f60f420 con 0x55965f6beab0 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): my global_id is 4233 2024-09-15T20:31:44.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55965f6259d0 con 0x55965f6beab0 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 client.?.objecter ms_handle_connect 0x55965f62d160 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4196128603 0 0) 0x7f040c002ff0 con 0x55965f629350 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x55965f60f420 con 0x55965f629350 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0418001a40 con 0x55965f629350 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): dump: 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.878 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4275535417 0 0) 0x55965f6259d0 con 0x55965f6beab0 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f040c002ff0 con 0x55965f6beab0 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0420001eb0 con 0x55965f6beab0 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): dump: 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.879 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3361841489 0 0) 0x55965f6bdf50 con 0x55965f62d160 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient(hunting): my global_id is 4240 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x55965f6259d0 con 0x55965f62d160 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3550521403 0 0) 0x55965f60f420 con 0x55965f629350 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 >> v1:172.21.15.146:6789/0 conn(0x55965f62d160 legacy=0x55965f6bd840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 >> v1:172.21.15.78:6789/0 conn(0x55965f6beab0 legacy=0x55965f6d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 monclient: found mon.c 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x55965f60e090 con 0x55965f629350 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 5 monclient: authenticate success, global_id 4238 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient: _renew_subs 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x55965f60e370 con 0x55965f629350 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:44.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient: _renew_subs 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x55965f60eaa0 con 0x55965f629350 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 1 librados: init done 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 librados: wait_for_osdmap waiting 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0418001cc0 con 0x55965f629350 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0418002360 con 0x55965f629350 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 monclient: dump: 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.882 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041f7fe640 4 set_mon_vals no callback set 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f0418002500 con 0x55965f629350 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(16..16 src has 1..16) ==== 3495+0+0 (unknown 988018965 0 0) 0x7f0418049920 con 0x55965f629350 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_map(16..16 src has 1..16) 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 3 client.4238.objecter handle_osd_map got epochs [16,16] > 0 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 3 client.4238.objecter handle_osd_map decoding full epoch 16 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 20 client.4238.objecter dump_active .. 0 homeless 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 librados: wait_for_osdmap done waiting 2024-09-15T20:31:44.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 start_mon_command cmd=[{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T20:31:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient: _send_command 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T20:31:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "osd pool application enable","pool": "rbd","app": "rbd"} v 0) -- 0x55965f6d9670 con 0x55965f629350 2024-09-15T20:31:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f042ce2e640 10 monclient: get_auth_request con 0x7f040c03c370 auth_method 0 2024-09-15T20:31:44.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:44.864+0000 7f041d7fa640 10 client.4238.objecter ms_handle_connect 0x7f040c03c370 2024-09-15T20:31:45.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.788+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}]=0 enabled application 'rbd' on pool 'rbd' v17) ==== 141+0+0 (unknown 1851909647 0 0) 0x7f04180153e0 con 0x55965f629350 2024-09-15T20:31:45.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.788+0000 7f041d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "osd pool application enable","pool": "rbd","app": "rbd"}] 2024-09-15T20:31:45.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.788+0000 7f041d7fa640 10 monclient: _finish_command 1 = system:0 enabled application 'rbd' on pool 'rbd' 2024-09-15T20:31:45.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.788+0000 7f042eb26100 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:45.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.788+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f0420001b20 con 0x55965f629350 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (unknown 391748328 0 0) 0x7f0418002ca0 con 0x55965f629350 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041d7fa640 10 monclient: handle_get_version_reply finishing 1 version 17 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041f7fe640 10 client.4238.objecter _get_latest_version latest 17, waiting 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041f7fe640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041f7fe640 10 monclient: _renew_subs 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041f7fe640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=17}) -- 0x7f0404003ca0 con 0x55965f629350 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 11 ==== osd_map(17..17 src has 1..17) ==== 658+0+0 (unknown 2463176763 0 0) 0x7f041801e270 con 0x55965f629350 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041d7fa640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_map(17..17 src has 1..17) 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041d7fa640 3 client.4238.objecter handle_osd_map got epochs [17,17] > 16 2024-09-15T20:31:45.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041d7fa640 3 client.4238.objecter handle_osd_map decoding incremental epoch 17 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041d7fa640 20 client.4238.objecter dump_active .. 0 homeless 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 10 librados: create oid=rbd_trash nspace= 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 10 client.4238.objecter _op_submit op 0x55965f6d98b0 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 20 client.4238.objecter _calc_target epoch 17 base rbd_trash @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 20 client.4238.objecter _calc_target target rbd_trash @2 -> pgid 2.5a749323 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 10 client.4238.objecter _calc_target raw pgid 2.5a749323 -> actual 2.3 acting [5,2] primary 5 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 20 client.4238.objecter _get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 10 client.4238.objecter _op_submit oid rbd_trash '@2' '@2' [create] tid 1 osd.5 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 20 client.4238.objecter get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 15 client.4238.objecter _session_op_assign 5 1 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 15 client.4238.objecter _send_op 1 to 2.3 on osd.5 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.146:6804/516207226 -- osd_op(unknown.0.0:1 2.3 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e17) -- 0x55965f6ddef0 con 0x55965f6da3e0 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:45.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042eb26100 5 client.4238.objecter 1 in flight 2024-09-15T20:31:45.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f042e891640 10 monclient: get_auth_request con 0x55965f6da3e0 auth_method 0 2024-09-15T20:31:45.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.792+0000 7f041d7fa640 10 client.4238.objecter ms_handle_connect 0x55965f6da3e0 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 1 -- 172.21.15.78:0/3300934940 <== osd.5 v1:172.21.15.146:6804/516207226 1 ==== osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) ==== 153+0+0 (unknown 1703302865 0 0) 0x7f0420004820 con 0x55965f6da3e0 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_op_reply(1 rbd_trash [create] v17'1 uv1 ondisk = 0) 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 10 client.4238.objecter in handle_osd_op_reply 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 7 client.4238.objecter handle_osd_op_reply 1 ondisk uv 1 in 2.3 attempt 0 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 10 client.4238.objecter op 0 rval 0 len 0 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 15 client.4238.objecter handle_osd_op_reply completed tid 1 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 15 client.4238.objecter _finish_op 1 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:45.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 15 client.4238.objecter _session_op_remove 5 1 2024-09-15T20:31:45.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042e891640 5 client.4238.objecter 0 in flight 2024-09-15T20:31:45.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.800+0000 7f042eb26100 10 librados: Objecter returned from create r=0 2024-09-15T20:31:45.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 10 client.4238.objecter _op_submit op 0x55965f730470 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 10 client.4238.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter _get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 10 client.4238.objecter _op_submit oid rbd_info '@2' '@2' [call rbd.metadata_list in=33b] tid 2 osd.5 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 15 client.4238.objecter _session_op_assign 5 2 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 15 client.4238.objecter _send_op 2 to 2.2 on osd.5 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.146:6804/516207226 -- osd_op(unknown.0.0:2 2.2 2:5cea7035:::rbd_info:head [call rbd.metadata_list in=33b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55965f7309c0 con 0x55965f6da3e0 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:45.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 5 client.4238.objecter 1 in flight 2024-09-15T20:31:45.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 1 -- 172.21.15.78:0/3300934940 <== osd.5 v1:172.21.15.146:6804/516207226 2 ==== osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4274215525 0 0) 0x7f0420004820 con 0x55965f6da3e0 2024-09-15T20:31:45.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_op_reply(2 rbd_info [call] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-15T20:31:45.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 10 client.4238.objecter in handle_osd_op_reply 2024-09-15T20:31:45.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 7 client.4238.objecter handle_osd_op_reply 2 ondisk uv 0 in 2.2 attempt 0 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 10 client.4238.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 10 client.4238.objecter op 0 rval 0 len 0 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 15 client.4238.objecter handle_osd_op_reply completed tid 2 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 15 client.4238.objecter _finish_op 2 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 15 client.4238.objecter _session_op_remove 5 2 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 5 client.4238.objecter 0 in flight 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 10 client.4238.objecter _op_submit op 0x55965f72f600 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter _calc_target epoch 17 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 10 client.4238.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter _get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 10 client.4238.objecter _op_submit oid rbd_info '@2' '@2' [read 0~0] tid 3 osd.5 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:45.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 15 client.4238.objecter _session_op_assign 5 3 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 15 client.4238.objecter _send_op 3 to 2.2 on osd.5 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.146:6804/516207226 -- osd_op(unknown.0.0:3 2.2 2:5cea7035:::rbd_info:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e17) -- 0x55965f72fab0 con 0x55965f6da3e0 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042eb26100 5 client.4238.objecter 1 in flight 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 1 -- 172.21.15.78:0/3300934940 <== osd.5 v1:172.21.15.146:6804/516207226 3 ==== osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 152+0+0 (unknown 4287864937 0 0) 0x7f0420004820 con 0x55965f6da3e0 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_op_reply(3 rbd_info [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 10 client.4238.objecter in handle_osd_op_reply 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 7 client.4238.objecter handle_osd_op_reply 3 ondisk uv 0 in 2.2 attempt 0 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 10 client.4238.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 10 client.4238.objecter op 0 rval 0 len 0 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 15 client.4238.objecter handle_osd_op_reply completed tid 3 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 15 client.4238.objecter _finish_op 3 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 15 client.4238.objecter _session_op_remove 5 3 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f042e891640 5 client.4238.objecter 0 in flight 2024-09-15T20:31:45.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f041effd640 10 client.4238.objecter allocate_selfmanaged_snap; pool: 2 2024-09-15T20:31:45.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f041effd640 10 client.4238.objecter pool_op_submit 4 2024-09-15T20:31:45.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f041effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:45.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:45.808+0000 7f041effd640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- pool_op(create unmanaged snap pool 2 tid 4 name v0) -- 0x7f04080030e0 con 0x55965f629350 2024-09-15T20:31:46.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 12 ==== pool_op_reply(tid 4 (0) Success v18) ==== 55+0+0 (unknown 3876692847 0 0) 0x7f04180197c0 con 0x55965f629350 2024-09-15T20:31:46.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 10 client.4238.objecter ms_dispatch 0x55965f510a30 pool_op_reply(tid 4 (0) Success v18) 2024-09-15T20:31:46.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 10 client.4238.objecter handle_pool_op_reply pool_op_reply(tid 4 (0) Success v18) 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 10 client.4238.objecter have request 4 at 0x7f0408003070 Op: create unmanaged snap 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 20 client.4238.objecter waiting for client to reach epoch 18 before calling back 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 10 monclient: _renew_subs 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=18}) -- 0x55965f60f420 con 0x55965f629350 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 10 client.4238.objecter done 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 13 ==== osd_map(18..18 src has 1..18) ==== 626+0+0 (unknown 3164432099 0 0) 0x7f041801e470 con 0x55965f629350 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_map(18..18 src has 1..18) 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 3 client.4238.objecter handle_osd_map got epochs [18,18] > 17 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 3 client.4238.objecter handle_osd_map decoding incremental epoch 18 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041d7fa640 20 client.4238.objecter dump_active .. 0 homeless 2024-09-15T20:31:46.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 20 librados: queue_aio_write 0x55965f72ba00 completion 0x55965f4f7ba0 write_seq 1 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 10 client.4238.objecter _op_submit op 0x7f0404006390 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 20 client.4238.objecter _calc_target epoch 18 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 20 client.4238.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 10 client.4238.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 20 client.4238.objecter _get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 10 client.4238.objecter _op_submit oid rbd_info '@2' '@2' [create,write 0~8 in=8b] tid 5 osd.5 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 20 client.4238.objecter get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 15 client.4238.objecter _session_op_assign 5 5 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 15 client.4238.objecter _send_op 5 to 2.2 on osd.5 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.146:6804/516207226 -- osd_op(unknown.0.0:5 2.2 2:5cea7035:::rbd_info:head [create,write 0~8 in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e18) -- 0x7f04040069d0 con 0x55965f6da3e0 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:46.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.812+0000 7f041f7fe640 5 client.4238.objecter 1 in flight 2024-09-15T20:31:46.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 1 -- 172.21.15.78:0/3300934940 <== osd.5 v1:172.21.15.146:6804/516207226 4 ==== osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) ==== 194+0+0 (unknown 2910594948 0 0) 0x7f0420004820 con 0x55965f6da3e0 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_op_reply(5 rbd_info [create,write 0~8] v18'1 uv1 ondisk = 0) 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 10 client.4238.objecter in handle_osd_op_reply 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 7 client.4238.objecter handle_osd_op_reply 5 ondisk uv 1 in 2.2 attempt 0 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 10 client.4238.objecter op 0 rval 0 len 0 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 10 client.4238.objecter op 1 rval 0 len 0 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 15 client.4238.objecter handle_osd_op_reply completed tid 5 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 15 client.4238.objecter _finish_op 5 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 15 client.4238.objecter _session_op_remove 5 5 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 5 client.4238.objecter 0 in flight 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f042e891640 20 librados: complete_aio_write 0x55965f4f7ba0 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f041effd640 10 client.4238.objecter delete_selfmanaged_snap; pool: 2; snap: 2 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f041effd640 10 client.4238.objecter pool_op_submit 6 2024-09-15T20:31:46.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f041effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:46.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:46.820+0000 7f041effd640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- pool_op(delete unmanaged snap pool 2 tid 6 name v18) -- 0x7f0408004550 con 0x55965f629350 2024-09-15T20:31:47.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 14 ==== pool_op_reply(tid 6 (0) Success v19) ==== 43+0+0 (unknown 500495984 0 0) 0x7f041801e700 con 0x55965f629350 2024-09-15T20:31:47.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 10 client.4238.objecter ms_dispatch 0x55965f510a30 pool_op_reply(tid 6 (0) Success v19) 2024-09-15T20:31:47.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 10 client.4238.objecter handle_pool_op_reply pool_op_reply(tid 6 (0) Success v19) 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 10 client.4238.objecter have request 6 at 0x7f0408003070 Op: delete unmanaged snap 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 20 client.4238.objecter waiting for client to reach epoch 19 before calling back 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 10 client.4238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 10 monclient: _renew_subs 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=19}) -- 0x7f040c002ff0 con 0x55965f629350 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 10 client.4238.objecter done 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 1 -- 172.21.15.78:0/3300934940 <== mon.2 v1:172.21.15.78:6790/0 15 ==== osd_map(19..19 src has 1..19) ==== 654+0+0 (unknown 3118647484 0 0) 0x7f0418014010 con 0x55965f629350 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_map(19..19 src has 1..19) 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 3 client.4238.objecter handle_osd_map got epochs [19,19] > 18 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 3 client.4238.objecter handle_osd_map decoding incremental epoch 19 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041d7fa640 20 client.4238.objecter dump_active .. 0 homeless 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 20 librados: queue_aio_write 0x55965f72ba00 completion 0x7f0408001d70 write_seq 2 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 10 client.4238.objecter _op_submit op 0x7f0404006390 2024-09-15T20:31:47.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 20 client.4238.objecter _calc_target epoch 19 base rbd_info @2 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 20 client.4238.objecter _calc_target target rbd_info @2 -> pgid 2.ac0e573a 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 10 client.4238.objecter _calc_target raw pgid 2.ac0e573a -> actual 2.2 acting [5,1] primary 5 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 20 client.4238.objecter _get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 10 client.4238.objecter _op_submit oid rbd_info '@2' '@2' [write 0~19 in=19b] tid 7 osd.5 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 20 client.4238.objecter get_session s=0x55965f6d9d60 osd=5 3 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 15 client.4238.objecter _session_op_assign 5 7 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 15 client.4238.objecter _send_op 7 to 2.2 on osd.5 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 1 -- 172.21.15.78:0/3300934940 --> v1:172.21.15.146:6804/516207226 -- osd_op(unknown.0.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19 in=19b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e19) -- 0x7f0404006db0 con 0x55965f6da3e0 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:47.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.828+0000 7f041f7fe640 5 client.4238.objecter 1 in flight 2024-09-15T20:31:47.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 1 -- 172.21.15.78:0/3300934940 <== osd.5 v1:172.21.15.146:6804/516207226 5 ==== osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) ==== 152+0+0 (unknown 900577457 0 0) 0x7f0420004820 con 0x55965f6da3e0 2024-09-15T20:31:47.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 10 client.4238.objecter ms_dispatch 0x55965f510a30 osd_op_reply(7 rbd_info [write 0~19] v19'2 uv2 ondisk = 0) 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 10 client.4238.objecter in handle_osd_op_reply 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 7 client.4238.objecter handle_osd_op_reply 7 ondisk uv 2 in 2.2 attempt 0 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 10 client.4238.objecter op 0 rval 0 len 0 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 15 client.4238.objecter handle_osd_op_reply completed tid 7 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 15 client.4238.objecter _finish_op 7 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 4 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 15 client.4238.objecter _session_op_remove 5 7 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 5 client.4238.objecter 0 in flight 2024-09-15T20:31:47.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 20 librados: complete_aio_write 0x7f0408001d70 2024-09-15T20:31:47.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 10 librados: watch_flush enter 2024-09-15T20:31:47.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 10 librados: watch_flush exit 2024-09-15T20:31:47.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 10 client.4238.objecter close_session for osd.5 2024-09-15T20:31:47.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 >> v1:172.21.15.146:6804/516207226 conn(0x55965f6da3e0 legacy=0x55965f6dc7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 20 client.4238.objecter put_session s=0x55965f6d9d60 osd=5 1 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 20 client.4238.objecter shutdown clearing up homeless session... 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 10 client.4238.objecter successfully canceled tick 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 >> v1:172.21.15.146:6812/435598642 conn(0x7f040c03c370 legacy=0x7f040c03e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 10 monclient: shutdown 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 >> v1:172.21.15.78:6790/0 conn(0x55965f629350 legacy=0x55965f6bd130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042e891640 1 -- 172.21.15.78:0/3300934940 reap_dead start 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 shutdown_connections 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 >> 172.21.15.78:0/3300934940 conn(0x55965f60f900 msgr2=0x55965f62ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 shutdown_connections 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 1 -- 172.21.15.78:0/3300934940 wait complete. 2024-09-15T20:31:47.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.836+0000 7f042eb26100 1 librados: shutdown 2024-09-15T20:31:47.846 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-09-15T20:31:47.846 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config log 1 --format=json 2024-09-15T20:31:47.847 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-15T20:31:47.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient: get_monmap_and_config 2024-09-15T20:31:47.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient: build_initial_monmap 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient: monmap: 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:47.973939+0000 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:47.973939+0000 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:47.975 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 1 Processor -- start 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 1 -- start start 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient: init 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf40745e0 con 0x7fcdf413c220 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf406ecf0 con 0x7fcdf40a24e0 2024-09-15T20:31:47.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf406efd0 con 0x7fcdf4135b60 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient(hunting): authenticate will time out at 1168.858753s 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcdf4135b60 0x7fcdf4135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39162/0 (socket says 172.21.15.78:39162) 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf2ffd640 1 -- 172.21.15.78:0/1757303752 learned_addr learned my addr 172.21.15.78:0/1757303752 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 527176025 0 0) 0x7fcdf406efd0 con 0x7fcdf4135b60 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient(hunting): my global_id is 4241 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcde0003530 con 0x7fcdf4135b60 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3528196669 0 0) 0x7fcdf40745e0 con 0x7fcdf413c220 2024-09-15T20:31:47.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient(hunting): my global_id is 4236 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdf406efd0 con 0x7fcdf413c220 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1048565243 0 0) 0x7fcde0003530 con 0x7fcdf4135b60 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 >> v1:172.21.15.146:6789/0 conn(0x7fcdf40a24e0 legacy=0x7fcdf413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 >> v1:172.21.15.78:6789/0 conn(0x7fcdf413c220 legacy=0x7fcdf413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 monclient: found mon.noname-b 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:47.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf4136650 con 0x7fcdf4135b60 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcdec001a20 con 0x7fcdf4135b60 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: dump: 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:47.981 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 5 monclient: authenticate success, global_id 4241 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcdec001bc0 con 0x7fcdf4135b60 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 1 -- 172.21.15.78:0/1757303752 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcdec001fd0 con 0x7fcdf4135b60 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf1ffb640 10 monclient: dump: 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:47.982 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.972+0000 7fcdf99d1640 4 set_mon_vals no callback set 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: shutdown 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- 172.21.15.78:0/1757303752 >> v1:172.21.15.78:6790/0 conn(0x7fcdf4135b60 legacy=0x7fcdf4135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:47.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- 172.21.15.78:0/1757303752 shutdown_connections 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- 172.21.15.78:0/1757303752 >> 172.21.15.78:0/1757303752 conn(0x7fcdf40a63c0 msgr2=0x7fcdf40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- 172.21.15.78:0/1757303752 shutdown_connections 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- 172.21.15.78:0/1757303752 wait complete. 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: build_initial_monmap 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: monmap: 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:47.984 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:47.985 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:47.985 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 librados: starting msgr at 2024-09-15T20:31:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 librados: starting objecter 2024-09-15T20:31:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 Processor -- start 2024-09-15T20:31:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- start start 2024-09-15T20:31:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 librados: setting wanted keys 2024-09-15T20:31:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 librados: calling monclient init 2024-09-15T20:31:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: init 2024-09-15T20:31:47.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient(hunting): picked mon.noname-b con 0x7fcdf40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient(hunting): picked mon.noname-c con 0x7fcdf4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient(hunting): picked mon.noname-a con 0x7fcdf413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf4072b10 con 0x7fcdf413c220 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf413f9b0 con 0x7fcdf40a24e0 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdf409d180 con 0x7fcdf4135b60 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient(hunting): authenticate will time out at 1168.861792s 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf27fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcdf4135b60 0x7fcdf4072400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39164/0 (socket says 172.21.15.78:39164) 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcdf413c220 0x7fcdf41d4e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37914/0 (socket says 172.21.15.78:37914) 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf27fc640 1 -- 172.21.15.78:0/934721286 learned_addr learned my addr 172.21.15.78:0/934721286 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:47.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 client.?.objecter ms_handle_connect 0x7fcdf4135b60 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 client.?.objecter ms_handle_connect 0x7fcdf413c220 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559049991 0 0) 0x7fcdf409d180 con 0x7fcdf4135b60 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): my global_id is 4244 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdd4004ec0 con 0x7fcdf4135b60 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1338063842 0 0) 0x7fcdf4072b10 con 0x7fcdf413c220 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:47.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): my global_id is 4239 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdf409d180 con 0x7fcdf413c220 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 client.?.objecter ms_handle_connect 0x7fcdf40a24e0 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4237841887 0 0) 0x7fcdd4004ec0 con 0x7fcdf4135b60 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcdf4072b10 con 0x7fcdf4135b60 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcddc001a40 con 0x7fcdf4135b60 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): dump: 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.992 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3329594514 0 0) 0x7fcdf409d180 con 0x7fcdf413c220 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcdd4004ec0 con 0x7fcdf413c220 2024-09-15T20:31:47.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcde4001320 con 0x7fcdf413c220 2024-09-15T20:31:47.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:47.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:47.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): dump: 2024-09-15T20:31:47.994 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:47.994 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:47.994 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.994 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310276448 0 0) 0x7fcdf413f9b0 con 0x7fcdf40a24e0 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient(hunting): my global_id is 4246 2024-09-15T20:31:47.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdf409d180 con 0x7fcdf40a24e0 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 56666948 0 0) 0x7fcdf4072b10 con 0x7fcdf4135b60 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 >> v1:172.21.15.146:6789/0 conn(0x7fcdf40a24e0 legacy=0x7fcdf4071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 >> v1:172.21.15.78:6789/0 conn(0x7fcdf413c220 legacy=0x7fcdf41d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 monclient: found mon.c 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf413b7c0 con 0x7fcdf4135b60 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient: _finish_auth 0 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:47.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 5 monclient: authenticate success, global_id 4244 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: _renew_subs 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf409e690 con 0x7fcdf4135b60 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: _renew_subs 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcdf40735b0 con 0x7fcdf4135b60 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 librados: init done 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcddc001d40 con 0x7fcdf4135b60 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcddc003a70 con 0x7fcdf4135b60 2024-09-15T20:31:47.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf17fa640 4 set_mon_vals no callback set 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 10 monclient: dump: 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:47.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd8005090 con 0x7fcdf4135b60 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.976+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fcddc003c10 con 0x7fcdf4135b60 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.980+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7fcddc04b1d0 con 0x7fcdf4135b60 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.980+0000 7fcdebfff640 10 client.4244.objecter ms_dispatch 0x7fcdf4000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.980+0000 7fcdebfff640 3 client.4244.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.980+0000 7fcdebfff640 3 client.4244.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.980+0000 7fcdebfff640 20 client.4244.objecter dump_active .. 0 homeless 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.980+0000 7fcdf2ffd640 10 monclient: get_auth_request con 0x7fcdd403e1f0 auth_method 0 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.980+0000 7fcdebfff640 10 client.4244.objecter ms_handle_connect 0x7fcdd403e1f0 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.984+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fcddc016b70 con 0x7fcdf4135b60 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.984+0000 7fcdebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:47.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:47.984+0000 7fcdebfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.152+0000 7fcdf99d1640 10 start_mon_command cmd=[{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-15T20:31:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.152+0000 7fcdf99d1640 10 monclient: _send_command 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-15T20:31:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.152+0000 7fcdf99d1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.152+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "config log", "num": 1, "format": "json"} v 0) -- 0x7fcdd8005380 con 0x7fcdf4135b60 2024-09-15T20:31:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.152+0000 7fcdebfff640 1 -- 172.21.15.78:0/934721286 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "config log", "num": 1, "format": "json"}]=0 v5) ==== 86+0+159 (unknown 882900978 0 3874445100) 0x7fcddc016470 con 0x7fcdf4135b60 2024-09-15T20:31:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.152+0000 7fcdebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config log", "num": 1, "format": "json"}] 2024-09-15T20:31:48.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.152+0000 7fcdebfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:48.157 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 10 librados: watch_flush enter 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 10 librados: watch_flush exit 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 20 client.4244.objecter shutdown clearing up homeless session... 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 10 client.4244.objecter successfully canceled tick 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 >> v1:172.21.15.146:6812/435598642 conn(0x7fcdd403e1f0 legacy=0x7fcdd40406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 10 monclient: shutdown 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 >> v1:172.21.15.78:6790/0 conn(0x7fcdf4135b60 legacy=0x7fcdf4072400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 shutdown_connections 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 >> 172.21.15.78:0/934721286 conn(0x7fcdf40a63c0 msgr2=0x7fcdf412e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 shutdown_connections 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 1 -- 172.21.15.78:0/934721286 wait complete. 2024-09-15T20:31:48.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.156+0000 7fcdf99d1640 1 librados: shutdown 2024-09-15T20:31:48.173 INFO:teuthology.orchestra.run.smithi078.stdout:[{"version":5,"timestamp":"2024-09-15T20:31:37.592131+0000","name":"","changes":[{"name":"osd.5/osd_mclock_max_capacity_iops_ssd","new_value":"5757.006497"}]}] 2024-09-15T20:31:48.174 INFO:tasks.ceph_manager:config epoch is 5 2024-09-15T20:31:48.174 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-15T20:31:48.174 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-09-15T20:31:48.174 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-09-15T20:31:48.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.288+0000 7effeed7f640 10 monclient: get_monmap_and_config 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.288+0000 7effeed7f640 10 monclient: build_initial_monmap 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.288+0000 7effeed7f640 10 monclient: monmap: 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:48.292045+0000 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:48.292045+0000 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:48.293 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.288+0000 7effeed7f640 1 Processor -- start 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.288+0000 7effeed7f640 1 -- start start 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.288+0000 7effeed7f640 10 monclient: init 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): picked mon.noname-c con 0x7effe8134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): picked mon.noname-a con 0x7effe809f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): picked mon.noname-b con 0x7effe806e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe80a0470 con 0x7effe809f5a0 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe809feb0 con 0x7effe8134bb0 2024-09-15T20:31:48.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe809e4a0 con 0x7effe806e960 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): authenticate will time out at 1169.176625s 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effed31c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7effe806e960 0x7effe813d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39178/0 (socket says 172.21.15.78:39178) 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effed31c640 1 -- 172.21.15.78:0/2990281126 learned_addr learned my addr 172.21.15.78:0/2990281126 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe7fff640 1 --1- 172.21.15.78:0/2990281126 >> v1:172.21.15.78:6789/0 conn(0x7effe809f5a0 0x7effe81399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37924/0 (socket says 172.21.15.78:37924) 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 859556824 0 0) 0x7effe809e4a0 con 0x7effe806e960 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient(hunting): my global_id is 4247 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effd0003530 con 0x7effe806e960 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903054780 0 0) 0x7effdc0014a0 con 0x7effe809f5a0 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient(hunting): my global_id is 4242 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effe809e4a0 con 0x7effe809f5a0 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3733907378 0 0) 0x7effe809e4a0 con 0x7effe809f5a0 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 >> v1:172.21.15.78:6790/0 conn(0x7effe806e960 legacy=0x7effe813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 >> v1:172.21.15.146:6789/0 conn(0x7effe8134bb0 legacy=0x7effe8134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 monclient: found mon.noname-a 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe80a0ff0 con 0x7effe809f5a0 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 5 monclient: authenticate success, global_id 4242 2024-09-15T20:31:48.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7effdc001b50 con 0x7effe809f5a0 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: dump: 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.303 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7effdc001cf0 con 0x7effe809f5a0 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 4 set_mon_vals no callback set 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 1 -- 172.21.15.78:0/2990281126 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7effdc004b70 con 0x7effe809f5a0 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe77fe640 10 monclient: dump: 2024-09-15T20:31:48.304 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: shutdown 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- 172.21.15.78:0/2990281126 >> v1:172.21.15.78:6789/0 conn(0x7effe809f5a0 legacy=0x7effe81399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- 172.21.15.78:0/2990281126 shutdown_connections 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- 172.21.15.78:0/2990281126 >> 172.21.15.78:0/2990281126 conn(0x7effe80a63c0 msgr2=0x7effe80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:48.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- 172.21.15.78:0/2990281126 shutdown_connections 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- 172.21.15.78:0/2990281126 wait complete. 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: build_initial_monmap 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: monmap: 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 librados: starting msgr at 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 librados: starting objecter 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 Processor -- start 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- start start 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 librados: setting wanted keys 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 librados: calling monclient init 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: init 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): picked mon.noname-a con 0x7effe806e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): picked mon.noname-b con 0x7effe809f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): picked mon.noname-c con 0x7effe8134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe811ec30 con 0x7effe806e960 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe811d400 con 0x7effe809f5a0 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7effe811ef10 con 0x7effe8134bb0 2024-09-15T20:31:48.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effeed7f640 10 monclient(hunting): authenticate will time out at 1169.179700s 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effecb1b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7effe806e960 0x7effe811c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37930/0 (socket says 172.21.15.78:37930) 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effecb1b640 1 -- 172.21.15.78:0/710552771 learned_addr learned my addr 172.21.15.78:0/710552771 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe57fa640 10 client.?.objecter ms_handle_connect 0x7effe806e960 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe57fa640 10 client.?.objecter ms_handle_connect 0x7effe8134bb0 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.292+0000 7effe57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2508095150 0 0) 0x7effe811ef10 con 0x7effe8134bb0 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): my global_id is 4250 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effd4004ef0 con 0x7effe8134bb0 2024-09-15T20:31:48.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622596315 0 0) 0x7effe811ec30 con 0x7effe806e960 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): my global_id is 4245 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effe811ef10 con 0x7effe806e960 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 client.?.objecter ms_handle_connect 0x7effe809f5a0 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3813079083 0 0) 0x7effd4004ef0 con 0x7effe8134bb0 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7effe811ec30 con 0x7effe8134bb0 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7effd8001960 con 0x7effe8134bb0 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): dump: 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.309 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 159898999 0 0) 0x7effe811ef10 con 0x7effe806e960 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7effd4004ef0 con 0x7effe806e960 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7effe0001a00 con 0x7effe806e960 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:48.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): dump: 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3161875199 0 0) 0x7effe811d400 con 0x7effe809f5a0 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient(hunting): my global_id is 4252 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7effe811ef10 con 0x7effe809f5a0 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3353490346 0 0) 0x7effe811ec30 con 0x7effe8134bb0 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 >> v1:172.21.15.146:6789/0 conn(0x7effe809f5a0 legacy=0x7effe811ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 >> v1:172.21.15.78:6789/0 conn(0x7effe806e960 legacy=0x7effe811c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 monclient: found mon.c 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe80a44f0 con 0x7effe8134bb0 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:48.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 5 monclient: authenticate success, global_id 4250 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 10 monclient: _renew_subs 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7effe811e950 con 0x7effe8134bb0 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 10 client.4250.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 10 monclient: _renew_subs 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7effe81d6530 con 0x7effe8134bb0 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 1 librados: init done 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7effd80047f0 con 0x7effe8134bb0 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7effd8004b40 con 0x7effe8134bb0 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 monclient: dump: 2024-09-15T20:31:48.313 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe6ffd640 4 set_mon_vals no callback set 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effc8005090 con 0x7effe8134bb0 2024-09-15T20:31:48.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7effd8010bb0 con 0x7effe8134bb0 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7effd804afb0 con 0x7effe8134bb0 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 client.4250.objecter ms_dispatch 0x7effe8000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 3 client.4250.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 3 client.4250.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 20 client.4250.objecter dump_active .. 0 homeless 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effecb1b640 10 monclient: get_auth_request con 0x7effd403e290 auth_method 0 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.296+0000 7effe57fa640 10 client.4250.objecter ms_handle_connect 0x7effd403e290 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.300+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7effd8016970 con 0x7effe8134bb0 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.300+0000 7effe57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.300+0000 7effe57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:48.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.512+0000 7effeed7f640 10 start_mon_command cmd=[{"prefix": "mgr dump", "format": "json"}] 2024-09-15T20:31:48.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.512+0000 7effeed7f640 10 monclient: _send_command 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T20:31:48.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.512+0000 7effeed7f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:48.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.512+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7effc8005380 con 0x7effe8134bb0 2024-09-15T20:31:48.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.516+0000 7effe57fa640 1 -- 172.21.15.78:0/710552771 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+93713 (unknown 481731318 0 2935188353) 0x7effd8016270 con 0x7effe8134bb0 2024-09-15T20:31:48.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.516+0000 7effe57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mgr dump", "format": "json"}] 2024-09-15T20:31:48.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.516+0000 7effe57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:48.520 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:48.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.516+0000 7effeed7f640 10 librados: watch_flush enter 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 10 librados: watch_flush exit 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 20 client.4250.objecter shutdown clearing up homeless session... 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 10 client.4250.objecter successfully canceled tick 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 >> v1:172.21.15.146:6812/435598642 conn(0x7effd403e290 legacy=0x7effd4040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 10 monclient: shutdown 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 >> v1:172.21.15.78:6790/0 conn(0x7effe8134bb0 legacy=0x7effe81d4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 shutdown_connections 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 >> 172.21.15.78:0/710552771 conn(0x7effe80a63c0 msgr2=0x7effe8072e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 shutdown_connections 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 1 -- 172.21.15.78:0/710552771 wait complete. 2024-09-15T20:31:48.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.520+0000 7effeed7f640 1 librados: shutdown 2024-09-15T20:31:48.536 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":5,"flags":0,"active_gid":4104,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6812","nonce":435598642}]},"active_addr":"172.21.15.146:6812/435598642","active_change":"2024-09-15T20:31:32.629937+0000","active_mgr_features":4540719139924082687,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to, use commas to separate multiple","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:sclorg-nginx-126","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.17","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_oauth2_proxy":{"name":"container_image_oauth2_proxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/oauth2-proxy/oauth2-proxy:v7.6.0","min":"","max":"","enum_allowed":[],"desc":"oauth2-proxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba_metrics":{"name":"container_image_samba_metrics","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-metrics:latest","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB metrics exporter container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_count_max":{"name":"ssh_keepalive_count_max","type":"int","level":"advanced","flags":0,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"How many times ssh connections can fail liveness checks before the host is marked offline","long_desc":"","tags":[],"see_also":[]},"ssh_keepalive_interval":{"name":"ssh_keepalive_interval","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"How often ssh connections are checked for liveness","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_requests":{"name":"max_requests","type":"int","level":"advanced","flags":0,"default_value":"500","min":"","max":"","enum_allowed":[],"desc":"Maximum number of requests to keep in memory. When new request comes in, the oldest request will be removed if the number of requests exceeds the max request number.if un-finished request is removed, error message will be logged in the ceph-mgr log.","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2662231271}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":60879583}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":238808815}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.146:0","nonce":2626167049}]}]} 2024-09-15T20:31:48.540 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-09-15T20:31:48.540 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-09-15T20:31:48.540 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:31:48.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient: get_monmap_and_config 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient: build_initial_monmap 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient: monmap: 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:48.651103+0000 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:48.651103+0000 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:48.652 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 1 Processor -- start 2024-09-15T20:31:48.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 1 -- start start 2024-09-15T20:31:48.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient: init 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient(hunting): picked mon.noname-a con 0x7f73e0135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient(hunting): picked mon.noname-c con 0x7f73e00a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient(hunting): picked mon.noname-b con 0x7f73e013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e00745e0 con 0x7f73e0135b60 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e006ecf0 con 0x7f73e00a24e0 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e006efd0 con 0x7f73e013c220 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73e6324640 10 monclient(hunting): authenticate will time out at 1169.535681s 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73dffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f73e0135b60 0x7f73e0135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37940/0 (socket says 172.21.15.78:37940) 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.648+0000 7f73dffff640 1 -- 172.21.15.78:0/900281721 learned_addr learned my addr 172.21.15.78:0/900281721 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 71444114 0 0) 0x7f73e00745e0 con 0x7f73e0135b60 2024-09-15T20:31:48.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient(hunting): my global_id is 4248 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73cc003530 con 0x7f73e0135b60 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 187756058 0 0) 0x7f73e006efd0 con 0x7f73e013c220 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient(hunting): my global_id is 4253 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73e00745e0 con 0x7f73e013c220 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3734534509 0 0) 0x7f73cc003530 con 0x7f73e0135b60 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 >> v1:172.21.15.78:6790/0 conn(0x7f73e013c220 legacy=0x7f73e013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 >> v1:172.21.15.146:6789/0 conn(0x7f73e00a24e0 legacy=0x7f73e013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 monclient: found mon.noname-a 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e0136650 con 0x7f73e0135b60 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 5 monclient: authenticate success, global_id 4248 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f73d00016a0 con 0x7f73e0135b60 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: dump: 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.659 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73d00018c0 con 0x7f73e0135b60 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 1 -- 172.21.15.78:0/900281721 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f73d0001cd0 con 0x7f73e0135b60 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 4 set_mon_vals no callback set 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73df7fe640 10 monclient: dump: 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.660 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: shutdown 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- 172.21.15.78:0/900281721 >> v1:172.21.15.78:6789/0 conn(0x7f73e0135b60 legacy=0x7f73e0135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- 172.21.15.78:0/900281721 shutdown_connections 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- 172.21.15.78:0/900281721 >> 172.21.15.78:0/900281721 conn(0x7f73e00a63c0 msgr2=0x7f73e00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:48.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- 172.21.15.78:0/900281721 shutdown_connections 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- 172.21.15.78:0/900281721 wait complete. 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: build_initial_monmap 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: monmap: 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 librados: starting msgr at 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 librados: starting objecter 2024-09-15T20:31:48.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 Processor -- start 2024-09-15T20:31:48.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- start start 2024-09-15T20:31:48.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 librados: setting wanted keys 2024-09-15T20:31:48.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 librados: calling monclient init 2024-09-15T20:31:48.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: init 2024-09-15T20:31:48.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:48.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:48.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient(hunting): picked mon.noname-c con 0x7f73e00a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:48.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient(hunting): picked mon.noname-a con 0x7f73e0135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient(hunting): picked mon.noname-b con 0x7f73e013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e0138750 con 0x7f73e0135b60 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e013bd50 con 0x7f73e013c220 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f73e009d180 con 0x7f73e00a24e0 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient(hunting): authenticate will time out at 1169.539017s 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dffff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f73e00a24e0 0x7f73e0071b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39204/0 (socket says 172.21.15.78:39204) 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dffff640 1 -- 172.21.15.78:0/2757432365 learned_addr learned my addr 172.21.15.78:0/2757432365 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 client.?.objecter ms_handle_connect 0x7f73e0135b60 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 client.?.objecter ms_handle_connect 0x7f73e00a24e0 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3649267512 0 0) 0x7f73e0138750 con 0x7f73e0135b60 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): my global_id is 4251 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73c0004ec0 con 0x7f73e0135b60 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3063758972 0 0) 0x7f73e009d180 con 0x7f73e00a24e0 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): my global_id is 4256 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73e0138750 con 0x7f73e00a24e0 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 client.?.objecter ms_handle_connect 0x7f73e013c220 2024-09-15T20:31:48.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1487370100 0 0) 0x7f73c0004ec0 con 0x7f73e0135b60 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73e009d180 con 0x7f73e0135b60 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f73c80018c0 con 0x7f73e0135b60 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): dump: 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.669 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2777935125 0 0) 0x7f73e0138750 con 0x7f73e00a24e0 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f73c0004ec0 con 0x7f73e00a24e0 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f73d0001c10 con 0x7f73e00a24e0 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): dump: 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.670 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1243539437 0 0) 0x7f73e013bd50 con 0x7f73e013c220 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient(hunting): my global_id is 4258 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f73e0138750 con 0x7f73e013c220 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2198011748 0 0) 0x7f73e009d180 con 0x7f73e0135b60 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 >> v1:172.21.15.78:6790/0 conn(0x7f73e00a24e0 legacy=0x7f73e0071b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 >> v1:172.21.15.146:6789/0 conn(0x7f73e013c220 legacy=0x7f73e0072990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 monclient: found mon.a 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73e0073630 con 0x7f73e0135b60 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 5 monclient: authenticate success, global_id 4251 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: _renew_subs 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.652+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f73e0138470 con 0x7f73e0135b60 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 10 client.4251.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:48.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 10 monclient: _renew_subs 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f73e01d3c20 con 0x7f73e0135b60 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 1 librados: init done 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f73c8003600 con 0x7f73e0135b60 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f73c8003950 con 0x7f73e0135b60 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73deffd640 4 set_mon_vals no callback set 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 10 monclient: dump: 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.673 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f73c4005090 con 0x7f73e0135b60 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f73c800f9c0 con 0x7f73e0135b60 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f73c8049d50 con 0x7f73e0135b60 2024-09-15T20:31:48.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 10 client.4251.objecter ms_dispatch 0x7f73e0000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:48.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 3 client.4251.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:48.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 3 client.4251.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:48.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 20 client.4251.objecter dump_active .. 0 homeless 2024-09-15T20:31:48.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dffff640 10 monclient: get_auth_request con 0x7f73c003e300 auth_method 0 2024-09-15T20:31:48.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.656+0000 7f73dd7fa640 10 client.4251.objecter ms_handle_connect 0x7f73c003e300 2024-09-15T20:31:48.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.660+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f73c80156f0 con 0x7f73e0135b60 2024-09-15T20:31:48.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.660+0000 7f73dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.660+0000 7f73dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:48.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.819+0000 7f73e6324640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:48.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.819+0000 7f73e6324640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:48.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.819+0000 7f73e6324640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.819+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f73c4005380 con 0x7f73e0135b60 2024-09-15T20:31:48.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73dd7fa640 1 -- 172.21.15.78:0/2757432365 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10937 (unknown 941590788 0 2001045350) 0x7f73c8019ad0 con 0x7f73e0135b60 2024-09-15T20:31:48.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:48.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:48.827 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:48.827 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":19,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","created":"2024-09-15T20:31:27.702188+0000","modified":"2024-09-15T20:31:47.806549+0000","last_up_change":"2024-09-15T20:31:37.715594+0000","last_in_change":"2024-09-15T20:31:29.437051+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:31:38.725860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T20:31:44.650592+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9003b16b-6a87-4659-8436-ff541ea8dfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6804","nonce":3648339548}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6805","nonce":3648339548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6807","nonce":3648339548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6806","nonce":3648339548}]},"public_addr":"172.21.15.78:6804/3648339548","cluster_addr":"172.21.15.78:6805/3648339548","heartbeat_back_addr":"172.21.15.78:6807/3648339548","heartbeat_front_addr":"172.21.15.78:6806/3648339548","state":["exists","up"]},{"osd":1,"uuid":"be602c68-218b-4573-8d19-eef568af9d4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6800","nonce":4006351464}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6801","nonce":4006351464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6803","nonce":4006351464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6802","nonce":4006351464}]},"public_addr":"172.21.15.78:6800/4006351464","cluster_addr":"172.21.15.78:6801/4006351464","heartbeat_back_addr":"172.21.15.78:6803/4006351464","heartbeat_front_addr":"172.21.15.78:6802/4006351464","state":["exists","up"]},{"osd":2,"uuid":"1fab92e2-4880-4f66-86f9-1c59752850a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6808","nonce":404661851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6809","nonce":404661851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6811","nonce":404661851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6810","nonce":404661851}]},"public_addr":"172.21.15.78:6808/404661851","cluster_addr":"172.21.15.78:6809/404661851","heartbeat_back_addr":"172.21.15.78:6811/404661851","heartbeat_front_addr":"172.21.15.78:6810/404661851","state":["exists","up"]},{"osd":3,"uuid":"74c2a9ff-0635-4664-9dc5-d156f4b60bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6800","nonce":490126554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6801","nonce":490126554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6803","nonce":490126554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6802","nonce":490126554}]},"public_addr":"172.21.15.146:6800/490126554","cluster_addr":"172.21.15.146:6801/490126554","heartbeat_back_addr":"172.21.15.146:6803/490126554","heartbeat_front_addr":"172.21.15.146:6802/490126554","state":["exists","up"]},{"osd":4,"uuid":"0abc805d-20bc-40e6-a25a-0d9694ace9eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6808","nonce":2960966471}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6809","nonce":2960966471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6811","nonce":2960966471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6810","nonce":2960966471}]},"public_addr":"172.21.15.146:6808/2960966471","cluster_addr":"172.21.15.146:6809/2960966471","heartbeat_back_addr":"172.21.15.146:6811/2960966471","heartbeat_front_addr":"172.21.15.146:6810/2960966471","state":["exists","up"]},{"osd":5,"uuid":"08c34b4f-deb1-401a-aab7-00854ae6712f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6804","nonce":516207226}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6805","nonce":516207226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6807","nonce":516207226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6806","nonce":516207226}]},"public_addr":"172.21.15.146:6804/516207226","cluster_addr":"172.21.15.146:6805/516207226","heartbeat_back_addr":"172.21.15.146:6807/516207226","heartbeat_front_addr":"172.21.15.146:6806/516207226","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.779852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.576609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.081865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.582230+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.088303+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.878442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 10 librados: watch_flush enter 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 10 librados: watch_flush exit 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 20 client.4251.objecter shutdown clearing up homeless session... 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 10 client.4251.objecter successfully canceled tick 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 >> v1:172.21.15.146:6812/435598642 conn(0x7f73c003e300 legacy=0x7f73c00407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 10 monclient: shutdown 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 >> v1:172.21.15.78:6789/0 conn(0x7f73e0135b60 legacy=0x7f73e0072280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 shutdown_connections 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 >> 172.21.15.78:0/2757432365 conn(0x7f73e00a63c0 msgr2=0x7f73e00a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 shutdown_connections 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 1 -- 172.21.15.78:0/2757432365 wait complete. 2024-09-15T20:31:48.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.823+0000 7f73e6324640 1 librados: shutdown 2024-09-15T20:31:48.845 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-09-15T20:31:48.845 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient: get_monmap_and_config 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient: build_initial_monmap 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient: monmap: 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:48.951430+0000 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:48.951430+0000 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:48.952 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.953 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:48.953 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:48.953 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:48.953 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 1 Processor -- start 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 1 -- start start 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient: init 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient(hunting): picked mon.noname-c con 0x7f1248135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient(hunting): picked mon.noname-a con 0x7f12480a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient(hunting): picked mon.noname-b con 0x7f124813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12480745e0 con 0x7f12480a24e0 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124806ecf0 con 0x7f1248135b60 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124806efd0 con 0x7f124813c220 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.947+0000 7f124f5bc640 10 monclient(hunting): authenticate will time out at 1169.835992s 2024-09-15T20:31:48.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124db59640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f124813c220 0x7f124813e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39220/0 (socket says 172.21.15.78:39220) 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124db59640 1 -- 172.21.15.78:0/3405329125 learned_addr learned my addr 172.21.15.78:0/3405329125 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035920986 0 0) 0x7f124806efd0 con 0x7f124813c220 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient(hunting): my global_id is 4259 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1234003530 con 0x7f124813c220 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3786250871 0 0) 0x7f12480745e0 con 0x7f12480a24e0 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient(hunting): my global_id is 4254 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f124806efd0 con 0x7f12480a24e0 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2123328811 0 0) 0x7f1234003530 con 0x7f124813c220 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 >> v1:172.21.15.146:6789/0 conn(0x7f1248135b60 legacy=0x7f1248135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 >> v1:172.21.15.78:6789/0 conn(0x7f12480a24e0 legacy=0x7f124813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 monclient: found mon.noname-b 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1248136650 con 0x7f124813c220 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: _finish_auth 0 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1240001a20 con 0x7f124813c220 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: dump: 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 5 monclient: authenticate success, global_id 4259 2024-09-15T20:31:48.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1240001c40 con 0x7f124813c220 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 4 set_mon_vals no callback set 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 1 -- 172.21.15.78:0/3405329125 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1240002050 con 0x7f124813c220 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123ffff640 10 monclient: dump: 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.960 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: shutdown 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- 172.21.15.78:0/3405329125 >> v1:172.21.15.78:6790/0 conn(0x7f124813c220 legacy=0x7f124813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- 172.21.15.78:0/3405329125 shutdown_connections 2024-09-15T20:31:48.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- 172.21.15.78:0/3405329125 >> 172.21.15.78:0/3405329125 conn(0x7f12480a63c0 msgr2=0x7f12480a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- 172.21.15.78:0/3405329125 shutdown_connections 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- 172.21.15.78:0/3405329125 wait complete. 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: build_initial_monmap 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: monmap: 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 librados: starting msgr at 2024-09-15T20:31:48.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 librados: starting objecter 2024-09-15T20:31:48.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 Processor -- start 2024-09-15T20:31:48.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- start start 2024-09-15T20:31:48.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 librados: setting wanted keys 2024-09-15T20:31:48.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 librados: calling monclient init 2024-09-15T20:31:48.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: init 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient(hunting): picked mon.noname-a con 0x7f12480a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient(hunting): picked mon.noname-b con 0x7f1248135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient(hunting): picked mon.noname-c con 0x7f124813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124809f2c0 con 0x7f12480a24e0 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f124809e600 con 0x7f1248135b60 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12480737f0 con 0x7f124813c220 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient(hunting): authenticate will time out at 1169.838907s 2024-09-15T20:31:48.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124d358640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f12480a24e0 0x7f1248072000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37976/0 (socket says 172.21.15.78:37976) 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124d358640 1 -- 172.21.15.78:0/3025674791 learned_addr learned my addr 172.21.15.78:0/3025674791 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 client.?.objecter ms_handle_connect 0x7f12480a24e0 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 client.?.objecter ms_handle_connect 0x7f124813c220 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460915552 0 0) 0x7f124806ecf0 con 0x7f12480a24e0 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): my global_id is 4257 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f122c004ec0 con 0x7f12480a24e0 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405525706 0 0) 0x7f12480737f0 con 0x7f124813c220 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:48.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): my global_id is 4262 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f124806ecf0 con 0x7f124813c220 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 975968279 0 0) 0x7f122c004ec0 con 0x7f12480a24e0 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12480737f0 con 0x7f12480a24e0 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 client.?.objecter ms_handle_connect 0x7f1248135b60 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1238001a40 con 0x7f12480a24e0 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): dump: 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.968 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 405371330 0 0) 0x7f124806ecf0 con 0x7f124813c220 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f122c004ec0 con 0x7f124813c220 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1240001df0 con 0x7f124813c220 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient(hunting): dump: 2024-09-15T20:31:48.969 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2980213598 0 0) 0x7f12480737f0 con 0x7f12480a24e0 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 >> v1:172.21.15.78:6790/0 conn(0x7f124813c220 legacy=0x7f12481d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 >> v1:172.21.15.146:6789/0 conn(0x7f1248135b60 legacy=0x7f1248072710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 monclient: found mon.a 2024-09-15T20:31:48.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1248072e20 con 0x7f12480a24e0 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 5 monclient: authenticate success, global_id 4257 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: _renew_subs 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f124809e8e0 con 0x7f12480a24e0 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: _renew_subs 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f124813fa80 con 0x7f12480a24e0 2024-09-15T20:31:48.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f124f5bc640 1 librados: init done 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1238001cc0 con 0x7f12480a24e0 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1238003a70 con 0x7f12480a24e0 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123f7fe640 4 set_mon_vals no callback set 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.951+0000 7f123dffb640 10 monclient: dump: 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:48.972 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f124f5bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f124f5bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f124f5bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1228005090 con 0x7f12480a24e0 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1238003c10 con 0x7f12480a24e0 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f123804b230 con 0x7f12480a24e0 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f123dffb640 10 client.4257.objecter ms_dispatch 0x7f1248000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f123dffb640 3 client.4257.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f123dffb640 3 client.4257.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f123dffb640 20 client.4257.objecter dump_active .. 0 homeless 2024-09-15T20:31:48.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f124cb57640 10 monclient: get_auth_request con 0x7f122c03e080 auth_method 0 2024-09-15T20:31:48.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.955+0000 7f123dffb640 10 client.4257.objecter ms_handle_connect 0x7f122c03e080 2024-09-15T20:31:48.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.959+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f124809f2c0 con 0x7f12480a24e0 2024-09-15T20:31:48.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.959+0000 7f123dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:48.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:48.959+0000 7f123dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1228005380 con 0x7f12480a24e0 2024-09-15T20:31:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f123dffb640 1 -- 172.21.15.78:0/3025674791 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v19) ==== 74+0+10937 (unknown 941590788 0 2001045350) 0x7f1238016c50 con 0x7f12480a24e0 2024-09-15T20:31:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f123dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:49.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f123dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:49.120 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:49.120 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":19,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","created":"2024-09-15T20:31:27.702188+0000","modified":"2024-09-15T20:31:47.806549+0000","last_up_change":"2024-09-15T20:31:37.715594+0000","last_in_change":"2024-09-15T20:31:29.437051+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:31:38.725860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T20:31:44.650592+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9003b16b-6a87-4659-8436-ff541ea8dfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6804","nonce":3648339548}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6805","nonce":3648339548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6807","nonce":3648339548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6806","nonce":3648339548}]},"public_addr":"172.21.15.78:6804/3648339548","cluster_addr":"172.21.15.78:6805/3648339548","heartbeat_back_addr":"172.21.15.78:6807/3648339548","heartbeat_front_addr":"172.21.15.78:6806/3648339548","state":["exists","up"]},{"osd":1,"uuid":"be602c68-218b-4573-8d19-eef568af9d4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6800","nonce":4006351464}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6801","nonce":4006351464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6803","nonce":4006351464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6802","nonce":4006351464}]},"public_addr":"172.21.15.78:6800/4006351464","cluster_addr":"172.21.15.78:6801/4006351464","heartbeat_back_addr":"172.21.15.78:6803/4006351464","heartbeat_front_addr":"172.21.15.78:6802/4006351464","state":["exists","up"]},{"osd":2,"uuid":"1fab92e2-4880-4f66-86f9-1c59752850a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6808","nonce":404661851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6809","nonce":404661851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6811","nonce":404661851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6810","nonce":404661851}]},"public_addr":"172.21.15.78:6808/404661851","cluster_addr":"172.21.15.78:6809/404661851","heartbeat_back_addr":"172.21.15.78:6811/404661851","heartbeat_front_addr":"172.21.15.78:6810/404661851","state":["exists","up"]},{"osd":3,"uuid":"74c2a9ff-0635-4664-9dc5-d156f4b60bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6800","nonce":490126554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6801","nonce":490126554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6803","nonce":490126554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6802","nonce":490126554}]},"public_addr":"172.21.15.146:6800/490126554","cluster_addr":"172.21.15.146:6801/490126554","heartbeat_back_addr":"172.21.15.146:6803/490126554","heartbeat_front_addr":"172.21.15.146:6802/490126554","state":["exists","up"]},{"osd":4,"uuid":"0abc805d-20bc-40e6-a25a-0d9694ace9eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6808","nonce":2960966471}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6809","nonce":2960966471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6811","nonce":2960966471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6810","nonce":2960966471}]},"public_addr":"172.21.15.146:6808/2960966471","cluster_addr":"172.21.15.146:6809/2960966471","heartbeat_back_addr":"172.21.15.146:6811/2960966471","heartbeat_front_addr":"172.21.15.146:6810/2960966471","state":["exists","up"]},{"osd":5,"uuid":"08c34b4f-deb1-401a-aab7-00854ae6712f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6804","nonce":516207226}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6805","nonce":516207226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6807","nonce":516207226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6806","nonce":516207226}]},"public_addr":"172.21.15.146:6804/516207226","cluster_addr":"172.21.15.146:6805/516207226","heartbeat_back_addr":"172.21.15.146:6807/516207226","heartbeat_front_addr":"172.21.15.146:6806/516207226","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.779852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.576609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.081865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.582230+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.088303+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.878442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:31:49.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 10 librados: watch_flush enter 2024-09-15T20:31:49.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 10 librados: watch_flush exit 2024-09-15T20:31:49.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 20 client.4257.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 10 client.4257.objecter successfully canceled tick 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 >> v1:172.21.15.146:6812/435598642 conn(0x7f122c03e080 legacy=0x7f122c040540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 10 monclient: shutdown 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 >> v1:172.21.15.78:6789/0 conn(0x7f12480a24e0 legacy=0x7f1248072000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 shutdown_connections 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.115+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 >> 172.21.15.78:0/3025674791 conn(0x7f12480a63c0 msgr2=0x7f124812e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.119+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 shutdown_connections 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.119+0000 7f124f5bc640 1 -- 172.21.15.78:0/3025674791 wait complete. 2024-09-15T20:31:49.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.119+0000 7f124f5bc640 1 librados: shutdown 2024-09-15T20:31:49.137 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-09-15T20:31:49.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-09-15T20:31:49.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-09-15T20:31:49.138 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-09-15T20:31:49.139 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-09-15T20:31:49.139 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-09-15T20:31:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient: monmap: 2024-09-15T20:31:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.250 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.249512+0000 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.249512+0000 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 1 Processor -- start 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient: monmap: 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.250240+0000 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.250240+0000 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.252 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.253 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.253 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 1 -- start start 2024-09-15T20:31:49.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient: init 2024-09-15T20:31:49.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 1 Processor -- start 2024-09-15T20:31:49.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:49.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient(hunting): picked mon.noname-a con 0x7f4174137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 1 -- start start 2024-09-15T20:31:49.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient(hunting): picked mon.noname-b con 0x7f41740a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient(hunting): picked mon.noname-c con 0x7f417413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient: init 2024-09-15T20:31:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41740a6420 con 0x7f4174137ce0 2024-09-15T20:31:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41740a6700 con 0x7f417413e5f0 2024-09-15T20:31:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41740a2110 con 0x7f41740a9610 2024-09-15T20:31:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient(hunting): authenticate will time out at 1170.134254s 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4172ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f41740a9610 0x7f417413ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39230/0 (socket says 172.21.15.78:39230) 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4172ffd640 1 -- 172.21.15.78:0/2403218040 learned_addr learned my addr 172.21.15.78:0/2403218040 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41737fe640 1 --1- 172.21.15.78:0/2403218040 >> v1:172.21.15.78:6789/0 conn(0x7f4174137ce0 0x7f41741380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:37986/0 (socket says 172.21.15.78:37986) 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241228701 0 0) 0x7f41740a6420 con 0x7f4174137ce0 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient(hunting): my global_id is 4260 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4160003530 con 0x7f4174137ce0 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3297721509 0 0) 0x7f41740a2110 con 0x7f41740a9610 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient(hunting): my global_id is 4265 2024-09-15T20:31:49.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41740a6420 con 0x7f41740a9610 2024-09-15T20:31:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:49.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1311744178 0 0) 0x7f4160003530 con 0x7f4174137ce0 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 >> v1:172.21.15.78:6790/0 conn(0x7f41740a9610 legacy=0x7f417413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 >> v1:172.21.15.146:6789/0 conn(0x7f417413e5f0 legacy=0x7f41741409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 monclient: found mon.noname-a 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41740a4bc0 con 0x7f4174137ce0 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f41640015d0 con 0x7f4174137ce0 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: dump: 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 5 monclient: authenticate success, global_id 4260 2024-09-15T20:31:49.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f373a59c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f373c137ce0 0x7f373c1380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39244/0 (socket says 172.21.15.78:39244) 2024-09-15T20:31:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient(hunting): picked mon.noname-b con 0x7f373c137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f373a59c640 1 -- 172.21.15.78:0/2193303382 learned_addr learned my addr 172.21.15.78:0/2193303382 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient(hunting): picked mon.noname-c con 0x7f373c0a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient(hunting): picked mon.noname-a con 0x7f373c13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 1 -- 172.21.15.78:0/2193303382 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373c0a6420 con 0x7f373c13e5f0 2024-09-15T20:31:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4164001770 con 0x7f4174137ce0 2024-09-15T20:31:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 1 -- 172.21.15.78:0/2403218040 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4164001b80 con 0x7f4174137ce0 2024-09-15T20:31:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f41727fc640 10 monclient: dump: 2024-09-15T20:31:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.261 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f4179a76640 4 set_mon_vals no callback set 2024-09-15T20:31:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 1 -- 172.21.15.78:0/2193303382 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373c0a6700 con 0x7f373c0a9610 2024-09-15T20:31:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 1 -- 172.21.15.78:0/2193303382 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373c0a2110 con 0x7f373c137ce0 2024-09-15T20:31:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f3740e71640 10 monclient(hunting): authenticate will time out at 1170.135662s 2024-09-15T20:31:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 75430425 0 0) 0x7f373c0a2110 con 0x7f373c137ce0 2024-09-15T20:31:49.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f373959a640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f373959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f373959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f373959a640 10 monclient(hunting): my global_id is 4268 2024-09-15T20:31:49.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.247+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3724003530 con 0x7f373c137ce0 2024-09-15T20:31:49.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 120000364 0 0) 0x7f3724003530 con 0x7f373c137ce0 2024-09-15T20:31:49.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 >> v1:172.21.15.146:6789/0 conn(0x7f373c0a9610 legacy=0x7f373c13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 >> v1:172.21.15.78:6789/0 conn(0x7f373c13e5f0 legacy=0x7f373c1409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: shutdown 2024-09-15T20:31:49.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- 172.21.15.78:0/2403218040 >> v1:172.21.15.78:6789/0 conn(0x7f4174137ce0 legacy=0x7f41741380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 1 monclient: found mon.noname-b 2024-09-15T20:31:49.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c0a4bc0 con 0x7f373c137ce0 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f372c0016a0 con 0x7f373c137ce0 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: dump: 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.266 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 5 monclient: authenticate success, global_id 4268 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- 172.21.15.78:0/2403218040 shutdown_connections 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- 172.21.15.78:0/2403218040 >> 172.21.15.78:0/2403218040 conn(0x7f41740aa6c0 msgr2=0x7f41740aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f372c001900 con 0x7f373c137ce0 2024-09-15T20:31:49.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- 172.21.15.78:0/2403218040 shutdown_connections 2024-09-15T20:31:49.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 1 -- 172.21.15.78:0/2193303382 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f372c001cd0 con 0x7f373c137ce0 2024-09-15T20:31:49.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f373959a640 10 monclient: dump: 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 4 set_mon_vals no callback set 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- 172.21.15.78:0/2403218040 wait complete. 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: monmap: 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.270 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 10 monclient: shutdown 2024-09-15T20:31:49.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 -- 172.21.15.78:0/2193303382 >> v1:172.21.15.78:6790/0 conn(0x7f373c137ce0 legacy=0x7f373c1380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 librados: starting msgr at 2024-09-15T20:31:49.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 librados: starting objecter 2024-09-15T20:31:49.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 -- 172.21.15.78:0/2193303382 shutdown_connections 2024-09-15T20:31:49.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 Processor -- start 2024-09-15T20:31:49.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 -- 172.21.15.78:0/2193303382 >> 172.21.15.78:0/2193303382 conn(0x7f373c0aa6c0 msgr2=0x7f373c0aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- start start 2024-09-15T20:31:49.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 librados: setting wanted keys 2024-09-15T20:31:49.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 librados: calling monclient init 2024-09-15T20:31:49.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: init 2024-09-15T20:31:49.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:49.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient(hunting): picked mon.noname-a con 0x7f41740a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient(hunting): picked mon.noname-b con 0x7f4174137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient(hunting): picked mon.noname-c con 0x7f417413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41740a9270 con 0x7f41740a9610 2024-09-15T20:31:49.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417413bc00 con 0x7f4174137ce0 2024-09-15T20:31:49.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f417413bee0 con 0x7f417413e5f0 2024-09-15T20:31:49.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient(hunting): authenticate will time out at 1170.138268s 2024-09-15T20:31:49.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 -- 172.21.15.78:0/2193303382 shutdown_connections 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f41737fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f41740a9610 0x7f4174138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38012/0 (socket says 172.21.15.78:38012) 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f41737fe640 1 -- 172.21.15.78:0/2238144516 learned_addr learned my addr 172.21.15.78:0/2238144516 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 client.?.objecter ms_handle_connect 0x7f41740a9610 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 client.?.objecter ms_handle_connect 0x7f417413e5f0 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1875802152 0 0) 0x7f41740a4bc0 con 0x7f41740a9610 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): my global_id is 4266 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4154004ec0 con 0x7f41740a9610 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101289194 0 0) 0x7f417413bee0 con 0x7f417413e5f0 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): my global_id is 4271 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f41740a4bc0 con 0x7f417413e5f0 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 client.?.objecter ms_handle_connect 0x7f4174137ce0 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3039598346 0 0) 0x7f41740a4bc0 con 0x7f417413e5f0 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f417413bee0 con 0x7f417413e5f0 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f416c001bc0 con 0x7f417413e5f0 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): dump: 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.278 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2094345277 0 0) 0x7f4154004ec0 con 0x7f41740a9610 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f41740a4bc0 con 0x7f41740a9610 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453299315 0 0) 0x7f417413bc00 con 0x7f4174137ce0 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): my global_id is 4273 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4154004ec0 con 0x7f4174137ce0 2024-09-15T20:31:49.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4164001b00 con 0x7f41740a9610 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient(hunting): dump: 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 440655410 0 0) 0x7f417413bee0 con 0x7f417413e5f0 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 >> v1:172.21.15.146:6789/0 conn(0x7f4174137ce0 legacy=0x7f41741395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 >> v1:172.21.15.78:6789/0 conn(0x7f41740a9610 legacy=0x7f4174138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 monclient: found mon.c 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4174139cd0 con 0x7f417413e5f0 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f416bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 5 monclient: authenticate success, global_id 4271 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: _renew_subs 2024-09-15T20:31:49.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f417413c1c0 con 0x7f417413e5f0 2024-09-15T20:31:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: _renew_subs 2024-09-15T20:31:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f417413a710 con 0x7f417413e5f0 2024-09-15T20:31:49.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f4179a76640 1 librados: init done 2024-09-15T20:31:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 -- 172.21.15.78:0/2193303382 wait complete. 2024-09-15T20:31:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 10 monclient: monmap: 2024-09-15T20:31:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.283 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 librados: starting msgr at 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 librados: starting objecter 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 Processor -- start 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 -- start start 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 librados: setting wanted keys 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 1 librados: calling monclient init 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.251+0000 7f3740e71640 10 monclient: init 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient(hunting): picked mon.noname-a con 0x7f373c0a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient(hunting): picked mon.noname-c con 0x7f373c137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient(hunting): picked mon.noname-b con 0x7f373c13e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373c0745f0 con 0x7f373c0a9610 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373c0a14b0 con 0x7f373c13e5f0 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f373c0708e0 con 0x7f373c137ce0 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient(hunting): authenticate will time out at 1170.140557s 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f373a59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f373c0a9610 0x7f373c1d8680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38018/0 (socket says 172.21.15.78:38018) 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f373a59c640 1 -- 172.21.15.78:0/678406184 learned_addr learned my addr 172.21.15.78:0/678406184 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.?.objecter ms_handle_connect 0x7f373c0a9610 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.?.objecter ms_handle_connect 0x7f373c137ce0 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3010742041 0 0) 0x7f373c0745f0 con 0x7f373c0a9610 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): my global_id is 4269 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37240024b0 con 0x7f373c0a9610 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.?.objecter ms_handle_connect 0x7f373c13e5f0 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f416c0039a0 con 0x7f417413e5f0 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f416c003cf0 con 0x7f417413e5f0 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 monclient: dump: 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.287 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.288 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f4171ffb640 4 set_mon_vals no callback set 2024-09-15T20:31:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f4179a76640 10 client.4271.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f4179a76640 15 client.4271.objecter _session_command_op_assign -1 1 2024-09-15T20:31:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f4179a76640 20 client.4271.objecter _get_session osd=-1 returning homeless 2024-09-15T20:31:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f4179a76640 10 client.4271.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f4179a76640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f4158002920 con 0x7f417413e5f0 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3823580228 0 0) 0x7f373c0708e0 con 0x7f373c137ce0 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): my global_id is 4274 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f373c0745f0 con 0x7f373c137ce0 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1260998622 0 0) 0x7f373c0a14b0 con 0x7f373c13e5f0 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): my global_id is 4276 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f373c0708e0 con 0x7f373c13e5f0 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1884044142 0 0) 0x7f37240024b0 con 0x7f373c0a9610 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f373c0a14b0 con 0x7f373c0a9610 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f372c001660 con 0x7f373c0a9610 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): dump: 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.290 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2927222090 0 0) 0x7f373c0745f0 con 0x7f373c137ce0 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37240024b0 con 0x7f373c137ce0 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3484476281 0 0) 0x7f373c0a14b0 con 0x7f373c0a9610 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 >> v1:172.21.15.78:6790/0 conn(0x7f373c137ce0 legacy=0x7f373c1de710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 >> v1:172.21.15.146:6789/0 conn(0x7f373c13e5f0 legacy=0x7f373c1e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 monclient: found mon.a 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c0715b0 con 0x7f373c0a9610 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 5 monclient: authenticate success, global_id 4269 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient: _renew_subs 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f373c072960 con 0x7f373c0a9610 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f372c004c00 con 0x7f373c0a9610 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f372c004f50 con 0x7f373c0a9610 2024-09-15T20:31:49.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: dump: 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.293 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient: _renew_subs 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f373c0733a0 con 0x7f373c0a9610 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3738d99640 4 set_mon_vals no callback set 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 1 librados: init done 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 client.4269.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 15 client.4269.objecter _session_command_op_assign -1 1 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 20 client.4269.objecter _get_session osd=-1 returning homeless 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 client.4269.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f373c06a080 con 0x7f373c0a9610 2024-09-15T20:31:49.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f372c010fc0 con 0x7f373c0a9610 2024-09-15T20:31:49.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f416c00fd60 con 0x7f417413e5f0 2024-09-15T20:31:49.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f416c04a0c0 con 0x7f417413e5f0 2024-09-15T20:31:49.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 client.4271.objecter ms_dispatch 0x7f4174000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 3 client.4271.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 3 client.4271.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 client.4271.objecter checking command 1 2024-09-15T20:31:49.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 20 client.4271.objecter _get_session s=0x7f4154044830 osd=5 3 2024-09-15T20:31:49.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 20 client.4271.objecter put_session s=0x7f4154044830 osd=5 3 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 15 client.4271.objecter _session_command_op_remove -1 1 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 20 client.4271.objecter _get_session s=0x7f4154044830 osd=5 3 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 20 client.4271.objecter get_session s=0x7f4154044830 osd=5 3 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 15 client.4271.objecter _session_command_op_assign 5 1 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 20 client.4271.objecter put_session s=0x7f4154044830 osd=5 4 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 client.4271.objecter _send_command 1 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.146:6804/516207226 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4154048b90 con 0x7f4154044ee0 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 20 client.4271.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f416c04a490 con 0x7f417413e5f0 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f41737fe640 10 monclient: get_auth_request con 0x7f415403e330 auth_method 0 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f4172ffd640 10 monclient: get_auth_request con 0x7f4154044ee0 auth_method 0 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 client.4271.objecter ms_handle_connect 0x7f415403e330 2024-09-15T20:31:49.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f416bfff640 10 client.4271.objecter ms_handle_connect 0x7f4154044ee0 2024-09-15T20:31:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f372c04b380 con 0x7f373c0a9610 2024-09-15T20:31:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.4269.objecter ms_dispatch 0x7f373c000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 3 client.4269.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 3 client.4269.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.4269.objecter checking command 1 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 20 client.4269.objecter _get_session s=0x7f3724044880 osd=0 3 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 20 client.4269.objecter put_session s=0x7f3724044880 osd=0 3 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 15 client.4269.objecter _session_command_op_remove -1 1 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 20 client.4269.objecter _get_session s=0x7f3724044880 osd=0 3 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 20 client.4269.objecter get_session s=0x7f3724044880 osd=0 3 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 15 client.4269.objecter _session_command_op_assign 0 1 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 20 client.4269.objecter put_session s=0x7f3724044880 osd=0 4 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 client.4269.objecter _send_command 1 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6804/3648339548 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3724048be0 con 0x7f3724044f30 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 20 client.4269.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f372c04b580 con 0x7f373c0a9610 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f372affd640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f373ad9d640 10 monclient: get_auth_request con 0x7f3724044f30 auth_method 0 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.255+0000 7f3739d9b640 10 monclient: get_auth_request con 0x7f372403e410 auth_method 0 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f372affd640 10 client.4269.objecter ms_handle_connect 0x7f3724044f30 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f372affd640 10 client.4269.objecter ms_handle_connect 0x7f372403e410 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== osd.0 v1:172.21.15.78:6804/3648339548 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f3724048be0 con 0x7f3724044f30 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f372affd640 10 client.4269.objecter ms_dispatch 0x7f373c000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f372affd640 10 client.4269.objecter _finish_command 1 = system:0 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f372affd640 20 client.4269.objecter put_session s=0x7f3724044880 osd=0 4 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f372affd640 15 client.4269.objecter _session_command_op_remove 0 1 2024-09-15T20:31:49.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== osd.5 v1:172.21.15.146:6804/516207226 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f4154048b90 con 0x7f4154044ee0 2024-09-15T20:31:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f416bfff640 10 client.4271.objecter ms_dispatch 0x7f4174000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:31:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f416bfff640 10 client.4271.objecter _finish_command 1 = system:0 2024-09-15T20:31:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f416bfff640 20 client.4271.objecter put_session s=0x7f4154044830 osd=5 4 2024-09-15T20:31:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.259+0000 7f416bfff640 15 client.4271.objecter _session_command_op_remove 5 1 2024-09-15T20:31:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 10 client.4269.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T20:31:49.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 15 client.4269.objecter _session_command_op_assign -1 2 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 20 client.4269.objecter _get_session s=0x7f3724044880 osd=0 3 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 20 client.4269.objecter put_session s=0x7f3724044880 osd=0 3 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 20 client.4269.objecter _get_session s=0x7f3724044880 osd=0 3 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 15 client.4269.objecter _session_command_op_remove -1 2 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 20 client.4269.objecter get_session s=0x7f3724044880 osd=0 3 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 15 client.4269.objecter _session_command_op_assign 0 2 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 20 client.4269.objecter put_session s=0x7f3724044880 osd=0 4 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 10 client.4269.objecter _send_command 2 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 --> v1:172.21.15.78:6804/3648339548 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f37200028e0 con 0x7f3724044f30 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f372affd640 1 -- 172.21.15.78:0/678406184 <== osd.0 v1:172.21.15.78:6804/3648339548 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7f37200028e0 con 0x7f3724044f30 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f372affd640 10 client.4269.objecter ms_dispatch 0x7f373c000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f372affd640 10 client.4269.objecter _finish_command 2 = system:0 2024-09-15T20:31:49.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f372affd640 20 client.4269.objecter put_session s=0x7f3724044880 osd=0 4 2024-09-15T20:31:49.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f372affd640 15 client.4269.objecter _session_command_op_remove 0 2 2024-09-15T20:31:49.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 10 librados: watch_flush enter 2024-09-15T20:31:49.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 10 librados: watch_flush exit 2024-09-15T20:31:49.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 10 client.4269.objecter close_session for osd.0 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 >> v1:172.21.15.78:6804/3648339548 conn(0x7f3724044f30 legacy=0x7f3724047370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 20 client.4269.objecter put_session s=0x7f3724044880 osd=0 1 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 20 client.4269.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 10 client.4269.objecter successfully canceled tick 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 >> v1:172.21.15.146:6812/435598642 conn(0x7f372403e410 legacy=0x7f37240408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 10 monclient: shutdown 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 >> v1:172.21.15.78:6789/0 conn(0x7f373c0a9610 legacy=0x7f373c1d8680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f373ad9d640 1 -- 172.21.15.78:0/678406184 reap_dead start 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 shutdown_connections 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 >> 172.21.15.78:0/678406184 conn(0x7f373c0aa6c0 msgr2=0x7f373c0a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.271+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 shutdown_connections 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 10 client.4271.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 15 client.4271.objecter _session_command_op_assign -1 2 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 20 client.4271.objecter _get_session s=0x7f4154044830 osd=5 3 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 20 client.4271.objecter put_session s=0x7f4154044830 osd=5 3 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 20 client.4271.objecter _get_session s=0x7f4154044830 osd=5 3 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 15 client.4271.objecter _session_command_op_remove -1 2 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 20 client.4271.objecter get_session s=0x7f4154044830 osd=5 3 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 15 client.4271.objecter _session_command_op_assign 5 2 2024-09-15T20:31:49.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 20 client.4271.objecter put_session s=0x7f4154044830 osd=5 4 2024-09-15T20:31:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 10 client.4271.objecter _send_command 2 2024-09-15T20:31:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 --> v1:172.21.15.146:6804/516207226 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4158001250 con 0x7f4154044ee0 2024-09-15T20:31:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f3740e71640 1 -- 172.21.15.78:0/678406184 wait complete. 2024-09-15T20:31:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f3740e71640 1 librados: shutdown 2024-09-15T20:31:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f416bfff640 1 -- 172.21.15.78:0/2238144516 <== osd.5 v1:172.21.15.146:6804/516207226 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3543570501) 0x7f4158001250 con 0x7f4154044ee0 2024-09-15T20:31:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f416bfff640 10 client.4271.objecter ms_dispatch 0x7f4174000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:31:49.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f416bfff640 10 client.4271.objecter _finish_command 2 = system:0 2024-09-15T20:31:49.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f416bfff640 20 client.4271.objecter put_session s=0x7f4154044830 osd=5 4 2024-09-15T20:31:49.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f416bfff640 15 client.4271.objecter _session_command_op_remove 5 2 2024-09-15T20:31:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 10 librados: watch_flush enter 2024-09-15T20:31:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 10 librados: watch_flush exit 2024-09-15T20:31:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 10 client.4271.objecter close_session for osd.5 2024-09-15T20:31:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 >> v1:172.21.15.146:6804/516207226 conn(0x7f4154044ee0 legacy=0x7f4154047320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 20 client.4271.objecter put_session s=0x7f4154044830 osd=5 1 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 20 client.4271.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 10 client.4271.objecter successfully canceled tick 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 >> v1:172.21.15.146:6812/435598642 conn(0x7f415403e330 legacy=0x7f41540407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 10 monclient: shutdown 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 >> v1:172.21.15.78:6790/0 conn(0x7f417413e5f0 legacy=0x7f41741d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4173fff640 1 -- 172.21.15.78:0/2238144516 reap_dead start 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 shutdown_connections 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 >> 172.21.15.78:0/2238144516 conn(0x7f41740aa6c0 msgr2=0x7f4174073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 shutdown_connections 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 -- 172.21.15.78:0/2238144516 wait complete. 2024-09-15T20:31:49.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.275+0000 7f4179a76640 1 librados: shutdown 2024-09-15T20:31:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:51539607556 2024-09-15T20:31:49.309 INFO:teuthology.orchestra.run.smithi078.stdout:51539607557 2024-09-15T20:31:49.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T20:31:49.310 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:31:49.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient: monmap: 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.306907+0000 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.306907+0000 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 1 Processor -- start 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 1 -- start start 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient: init 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient(hunting): picked mon.noname-b con 0x7fdd24137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient(hunting): picked mon.noname-c con 0x7fdd240a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient(hunting): picked mon.noname-a con 0x7fdd2413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd240a6420 con 0x7fdd2413e5f0 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd240a6700 con 0x7fdd240a9610 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd240a2110 con 0x7fdd24137ce0 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2bbe2640 10 monclient(hunting): authenticate will time out at 1170.191252s 2024-09-15T20:31:49.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2997e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fdd24137ce0 0x7fdd241380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39278/0 (socket says 172.21.15.78:39278) 2024-09-15T20:31:49.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2997e640 1 -- 172.21.15.78:0/1423348104 learned_addr learned my addr 172.21.15.78:0/1423348104 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7f71b272f640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7f71b272f640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3117115912 0 0) 0x7fdd240a2110 con 0x7fdd24137ce0 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 10 monclient(hunting): my global_id is 4277 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd10003530 con 0x7fdd24137ce0 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258402921 0 0) 0x7fdd240a6420 con 0x7fdd2413e5f0 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 10 monclient(hunting): my global_id is 4272 2024-09-15T20:31:49.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd240a2110 con 0x7fdd2413e5f0 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7f71b272f640 10 monclient: monmap: 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.308134+0000 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.308134+0000 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.316 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 789497920 0 0) 0x7fdd10003530 con 0x7fdd24137ce0 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 >> v1:172.21.15.146:6789/0 conn(0x7fdd240a9610 legacy=0x7fdd2413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.303+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 >> v1:172.21.15.78:6789/0 conn(0x7fdd2413e5f0 legacy=0x7fdd241409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 1 monclient: found mon.noname-b 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd240a4bc0 con 0x7fdd24137ce0 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd200015d0 con 0x7fdd24137ce0 2024-09-15T20:31:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 5 monclient: authenticate success, global_id 4277 2024-09-15T20:31:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 Processor -- start 2024-09-15T20:31:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:49.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: dump: 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- start start 2024-09-15T20:31:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient: init 2024-09-15T20:31:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd200017f0 con 0x7fdd24137ce0 2024-09-15T20:31:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 1 -- 172.21.15.78:0/1423348104 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd20001bc0 con 0x7fdd24137ce0 2024-09-15T20:31:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 4 set_mon_vals no callback set 2024-09-15T20:31:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2897c640 10 monclient: dump: 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.321 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient: monmap: 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.309441+0000 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.309441+0000 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.322 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.323 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient: monmap: 2024-09-15T20:31:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.323 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.324 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.309525+0000 2024-09-15T20:31:49.324 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.309525+0000 2024-09-15T20:31:49.324 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.324 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.324 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.324 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.324 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.324 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:49.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient(hunting): picked mon.noname-c con 0x7f71ac09d490 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient(hunting): picked mon.noname-a con 0x7f71ac09df80 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient(hunting): picked mon.noname-b con 0x7f71ac09f930 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71ac09efa0 con 0x7f71ac09df80 2024-09-15T20:31:49.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71ac0a1c20 con 0x7f71ac09d490 2024-09-15T20:31:49.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71ac09ec50 con 0x7f71ac09f930 2024-09-15T20:31:49.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient(hunting): authenticate will time out at 1170.192847s 2024-09-15T20:31:49.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71ab7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f71ac09df80 0x7f71ac13cfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38038/0 (socket says 172.21.15.78:38038) 2024-09-15T20:31:49.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71ab7fe640 1 -- 172.21.15.78:0/2747066295 learned_addr learned my addr 172.21.15.78:0/2747066295 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient: shutdown 2024-09-15T20:31:49.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/1423348104 >> v1:172.21.15.78:6790/0 conn(0x7fdd24137ce0 legacy=0x7fdd241380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 603257039 0 0) 0x7f71ac09efa0 con 0x7f71ac09df80 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient(hunting): my global_id is 4275 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7194003530 con 0x7f71ac09df80 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1278256621 0 0) 0x7f71ac09ec50 con 0x7f71ac09f930 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient(hunting): my global_id is 4280 2024-09-15T20:31:49.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71ac09efa0 con 0x7f71ac09f930 2024-09-15T20:31:49.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 1 Processor -- start 2024-09-15T20:31:49.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/1423348104 shutdown_connections 2024-09-15T20:31:49.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/1423348104 >> 172.21.15.78:0/1423348104 conn(0x7fdd240aa6c0 msgr2=0x7fdd240aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 1 -- start start 2024-09-15T20:31:49.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient: init 2024-09-15T20:31:49.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1502560781 0 0) 0x7f7194003530 con 0x7f71ac09df80 2024-09-15T20:31:49.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 >> v1:172.21.15.78:6790/0 conn(0x7f71ac09f930 legacy=0x7f71ac1406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/1423348104 shutdown_connections 2024-09-15T20:31:49.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/1423348104 wait complete. 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 >> v1:172.21.15.146:6789/0 conn(0x7f71ac09d490 legacy=0x7f71ac09d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 monclient: found mon.noname-a 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ac0a37e0 con 0x7f71ac09df80 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f719c0016a0 con 0x7f71ac09df80 2024-09-15T20:31:49.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 1 Processor -- start 2024-09-15T20:31:49.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: dump: 2024-09-15T20:31:49.331 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.331 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.331 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 5 monclient: authenticate success, global_id 4275 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 1 -- start start 2024-09-15T20:31:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient: init 2024-09-15T20:31:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f719c0018c0 con 0x7f71ac09df80 2024-09-15T20:31:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 1 -- 172.21.15.78:0/2747066295 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f719c001d50 con 0x7f71ac09df80 2024-09-15T20:31:49.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 4 set_mon_vals no callback set 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71aaffd640 10 monclient: dump: 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.334 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient(hunting): picked mon.noname-b con 0x7fe6d0137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient(hunting): picked mon.noname-c con 0x7fe6d00a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient(hunting): picked mon.noname-a con 0x7fe6d013e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d00a6420 con 0x7fe6d013e5f0 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d00a6700 con 0x7fe6d00a9610 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d00a2110 con 0x7fe6d0137ce0 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient(hunting): authenticate will time out at 1170.194236s 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6cffff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe6d0137ce0 0x7fe6d01380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39304/0 (socket says 172.21.15.78:39304) 2024-09-15T20:31:49.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6cffff640 1 -- 172.21.15.78:0/2964154778 learned_addr learned my addr 172.21.15.78:0/2964154778 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:49.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ec137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ec0a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ec13e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec0a6420 con 0x7fe2ec137ce0 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec0a6700 con 0x7fe2ec0a9610 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec0a2110 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f4100640 10 monclient(hunting): authenticate will time out at 1170.194482s 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 459120416 0 0) 0x7fe6d00a2110 con 0x7fe6d0137ce0 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.338 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient(hunting): my global_id is 4283 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient: monmap: 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.339 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.340 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.340 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.340 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 librados: starting msgr at 2024-09-15T20:31:49.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 librados: starting objecter 2024-09-15T20:31:49.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6b8003530 con 0x7fe6d0137ce0 2024-09-15T20:31:49.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient: shutdown 2024-09-15T20:31:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- 172.21.15.78:0/2747066295 >> v1:172.21.15.78:6789/0 conn(0x7f71ac09df80 legacy=0x7f71ac13cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 Processor -- start 2024-09-15T20:31:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- 172.21.15.78:0/2747066295 shutdown_connections 2024-09-15T20:31:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- 172.21.15.78:0/2747066295 >> 172.21.15.78:0/2747066295 conn(0x7f71ac0a2a90 msgr2=0x7f71ac0a2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f269d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe2ec13e5f0 0x7fe2ec1409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39310/0 (socket says 172.21.15.78:39310) 2024-09-15T20:31:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f269d640 1 -- 172.21.15.78:0/1058965372 learned_addr learned my addr 172.21.15.78:0/1058965372 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- start start 2024-09-15T20:31:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 librados: setting wanted keys 2024-09-15T20:31:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 librados: calling monclient init 2024-09-15T20:31:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient: init 2024-09-15T20:31:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- 172.21.15.78:0/2747066295 shutdown_connections 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3301215205 0 0) 0x7fe6b8003530 con 0x7fe6d0137ce0 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 >> v1:172.21.15.146:6789/0 conn(0x7fe6d00a9610 legacy=0x7fe6d013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 >> v1:172.21.15.78:6789/0 conn(0x7fe6d013e5f0 legacy=0x7fe6d01409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 monclient: found mon.noname-b 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d00a4bc0 con 0x7fe6d0137ce0 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.343 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 1 -- 172.21.15.78:0/2747066295 wait complete. 2024-09-15T20:31:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 5 monclient: authenticate success, global_id 4283 2024-09-15T20:31:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.344 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe6c00016a0 con 0x7fe6d0137ce0 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: dump: 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.345 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6c0001900 con 0x7fe6d0137ce0 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 1 -- 172.21.15.78:0/2964154778 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe6c0001cd0 con 0x7fe6d0137ce0 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6ceffd640 10 monclient: dump: 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.346 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe6d68df640 4 set_mon_vals no callback set 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586989870 0 0) 0x7fe2ec0a2110 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): my global_id is 4286 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d8003530 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1539780986 0 0) 0x7fe2ec0a6420 con 0x7fe2ec137ce0 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): my global_id is 4281 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec0a2110 con 0x7fe2ec137ce0 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2476233335 0 0) 0x7fe2ec0a6700 con 0x7fe2ec0a9610 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient(hunting): my global_id is 4285 2024-09-15T20:31:49.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec0a6420 con 0x7fe2ec0a9610 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient(hunting): picked mon.noname-a con 0x7fdd240a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient(hunting): picked mon.noname-c con 0x7fdd24137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient(hunting): picked mon.noname-b con 0x7fdd2413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd2413dee0 con 0x7fdd240a9610 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd24139cd0 con 0x7fdd2413e5f0 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd2413a2a0 con 0x7fdd24137ce0 2024-09-15T20:31:49.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2bbe2640 10 monclient(hunting): authenticate will time out at 1170.195583s 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7f71b272f640 10 monclient: monmap: 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3245230508 0 0) 0x7fe2d8003530 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 >> v1:172.21.15.146:6789/0 conn(0x7fe2ec0a9610 legacy=0x7fe2ec13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 >> v1:172.21.15.78:6789/0 conn(0x7fe2ec137ce0 legacy=0x7fe2ec1380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 monclient: found mon.noname-b 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ec0a4bc0 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe2e4001a20 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fe2f0e9a640 10 monclient: dump: 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.353 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2997e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdd240a9610 0x7fdd24138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38052/0 (socket says 172.21.15.78:38052) 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd2997e640 1 -- 172.21.15.78:0/825771535 learned_addr learned my addr 172.21.15.78:0/825771535 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 10 client.?.objecter ms_handle_connect 0x7fdd24137ce0 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 10 client.?.objecter ms_handle_connect 0x7fdd240a9610 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 871868521 0 0) 0x7fdd2413a2a0 con 0x7fdd24137ce0 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.307+0000 7fdd1a7fc640 10 monclient(hunting): my global_id is 4289 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd08004ec0 con 0x7fdd24137ce0 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1379060280 0 0) 0x7fdd240a4bc0 con 0x7fdd240a9610 2024-09-15T20:31:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): my global_id is 4284 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd2413a2a0 con 0x7fdd240a9610 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 client.?.objecter ms_handle_connect 0x7fdd2413e5f0 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2e4001bc0 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f0e9a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f0e9a640 1 -- 172.21.15.78:0/1058965372 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe2e4001f90 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f0e9a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f0e9a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f0e9a640 10 monclient: dump: 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.356 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient: shutdown 2024-09-15T20:31:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- 172.21.15.78:0/2964154778 >> v1:172.21.15.78:6790/0 conn(0x7fe6d0137ce0 legacy=0x7fe6d01380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 librados: starting msgr at 2024-09-15T20:31:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 librados: starting objecter 2024-09-15T20:31:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 Processor -- start 2024-09-15T20:31:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 5 monclient: authenticate success, global_id 4286 2024-09-15T20:31:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 4 set_mon_vals no callback set 2024-09-15T20:31:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- 172.21.15.78:0/2964154778 shutdown_connections 2024-09-15T20:31:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- 172.21.15.78:0/2964154778 >> 172.21.15.78:0/2964154778 conn(0x7fe6d00aa6c0 msgr2=0x7fe6d00aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3055632896 0 0) 0x7fdd08004ec0 con 0x7fdd24137ce0 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd240a4bc0 con 0x7fdd24137ce0 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd14001940 con 0x7fdd24137ce0 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): dump: 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217154887 0 0) 0x7fdd24139cd0 con 0x7fdd2413e5f0 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient(hunting): my global_id is 4288 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd08004ec0 con 0x7fdd2413e5f0 2024-09-15T20:31:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3405063535 0 0) 0x7fdd240a4bc0 con 0x7fdd24137ce0 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 >> v1:172.21.15.146:6789/0 conn(0x7fdd2413e5f0 legacy=0x7fdd241d91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 >> v1:172.21.15.78:6789/0 conn(0x7fdd240a9610 legacy=0x7fdd24138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 monclient: found mon.c 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd2413c360 con 0x7fdd24137ce0 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 5 monclient: authenticate success, global_id 4289 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 10 monclient: _renew_subs 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd2413c640 con 0x7fdd24137ce0 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 10 monclient: _renew_subs 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd2413ab10 con 0x7fdd24137ce0 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 1 librados: init done 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd14003620 con 0x7fdd24137ce0 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd140039b0 con 0x7fdd24137ce0 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1b7fe640 4 set_mon_vals no callback set 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: dump: 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- 172.21.15.78:0/2964154778 shutdown_connections 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 10 monclient: shutdown 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 1 -- 172.21.15.78:0/1058965372 >> v1:172.21.15.78:6790/0 conn(0x7fe2ec13e5f0 legacy=0x7fe2ec1409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- 172.21.15.78:0/2964154778 wait complete. 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient: monmap: 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.366 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fdd14003bd0 con 0x7fdd24137ce0 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 -- start start 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 librados: setting wanted keys 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 librados: calling monclient init 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient: init 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient(hunting): picked mon.noname-c con 0x7f71ac09d490 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient(hunting): picked mon.noname-a con 0x7f71ac09df80 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient(hunting): picked mon.noname-b con 0x7f71ac09f930 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71ac06eba0 con 0x7f71ac09df80 2024-09-15T20:31:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71ac09d180 con 0x7f71ac09f930 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71ac11e0d0 con 0x7f71ac09d490 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient(hunting): authenticate will time out at 1170.197003s 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71abfff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f71ac09d490 0x7f71ac1d1320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39330/0 (socket says 172.21.15.78:39330) 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71abfff640 1 -- 172.21.15.78:0/1792947077 learned_addr learned my addr 172.21.15.78:0/1792947077 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 client.?.objecter ms_handle_connect 0x7f71ac09d490 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 491508737 0 0) 0x7f71ac11e0d0 con 0x7f71ac09d490 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient(hunting): my global_id is 4292 2024-09-15T20:31:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7198004ec0 con 0x7f71ac09d490 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3457147979 0 0) 0x7f7198004ec0 con 0x7f71ac09d490 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71ac11e0d0 con 0x7f71ac09d490 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71a0001a40 con 0x7f71ac09d490 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient(hunting): dump: 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1603088727 0 0) 0x7f71ac11e0d0 con 0x7f71ac09d490 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 10 client.4289.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 15 client.4289.objecter _session_command_op_assign -1 1 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 20 client.4289.objecter _get_session osd=-1 returning homeless 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 >> v1:172.21.15.146:6789/0 conn(0x7f71ac09f930 legacy=0x7f71ac1d9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 >> v1:172.21.15.78:6789/0 conn(0x7f71ac09df80 legacy=0x7f71ac1d6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 monclient: found mon.c 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71ac120430 con 0x7f71ac09d490 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fdd0c002920 con 0x7fdd24137ce0 2024-09-15T20:31:49.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 5 monclient: authenticate success, global_id 4292 2024-09-15T20:31:49.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient: _renew_subs 2024-09-15T20:31:49.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71ac11fb90 con 0x7f71ac09d490 2024-09-15T20:31:49.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient: _renew_subs 2024-09-15T20:31:49.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f71ac11ec30 con 0x7f71ac09d490 2024-09-15T20:31:49.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 librados: init done 2024-09-15T20:31:49.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 1 -- 172.21.15.78:0/1058965372 shutdown_connections 2024-09-15T20:31:49.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 1 -- 172.21.15.78:0/1058965372 >> 172.21.15.78:0/1058965372 conn(0x7fe2ec0aa6c0 msgr2=0x7fe2ec0aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 librados: starting msgr at 2024-09-15T20:31:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 librados: starting objecter 2024-09-15T20:31:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 Processor -- start 2024-09-15T20:31:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- start start 2024-09-15T20:31:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 librados: setting wanted keys 2024-09-15T20:31:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 librados: calling monclient init 2024-09-15T20:31:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient: init 2024-09-15T20:31:49.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 1 -- 172.21.15.78:0/1058965372 shutdown_connections 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7fdd14049f20 con 0x7fdd24137ce0 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 client.4289.objecter ms_dispatch 0x7fdd24000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 3 client.4289.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 3 client.4289.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 client.4289.objecter checking command 1 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 20 client.4289.objecter _get_session s=0x7fdd08044790 osd=4 3 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 20 client.4289.objecter put_session s=0x7fdd08044790 osd=4 3 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 15 client.4289.objecter _session_command_op_remove -1 1 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 20 client.4289.objecter _get_session s=0x7fdd08044790 osd=4 3 2024-09-15T20:31:49.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 20 client.4289.objecter get_session s=0x7fdd08044790 osd=4 3 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 15 client.4289.objecter _session_command_op_assign 4 1 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 20 client.4289.objecter put_session s=0x7fdd08044790 osd=4 4 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 client.4289.objecter _send_command 1 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.146:6808/2960966471 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd08048ad0 con 0x7fdd08044e40 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 20 client.4289.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fdd14004320 con 0x7fdd24137ce0 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2a17f640 10 monclient: get_auth_request con 0x7fdd08044e40 auth_method 0 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd2997e640 10 monclient: get_auth_request con 0x7fdd0803e2e0 auth_method 0 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71a00038d0 con 0x7f71ac09d490 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71a0003ca0 con 0x7f71ac09d490 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 client.4292.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 15 client.4292.objecter _session_command_op_assign -1 1 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 20 client.4292.objecter _get_session osd=-1 returning homeless 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71a8ff9640 10 monclient: dump: 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.377 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.378 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.378 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.378 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.378 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71aa7fc640 4 set_mon_vals no callback set 2024-09-15T20:31:49.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f71ac11ff70 con 0x7f71ac09d490 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient(hunting): picked mon.noname-b con 0x7fe6d00a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient(hunting): picked mon.noname-c con 0x7fe6d0137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient(hunting): picked mon.noname-a con 0x7fe6d013e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d00a3da0 con 0x7fe6d013e5f0 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d013c780 con 0x7fe6d00a9610 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6d00a14b0 con 0x7fe6d0137ce0 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6d68df640 10 monclient(hunting): authenticate will time out at 1170.199153s 2024-09-15T20:31:49.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 client.4289.objecter ms_handle_connect 0x7fdd08044e40 2024-09-15T20:31:49.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6cf7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe6d0137ce0 0x7fe6d01d1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39332/0 (socket says 172.21.15.78:39332) 2024-09-15T20:31:49.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6cf7fe640 1 -- 172.21.15.78:0/3340253074 learned_addr learned my addr 172.21.15.78:0/3340253074 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 1 -- 172.21.15.78:0/1058965372 wait complete. 2024-09-15T20:31:49.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fdd1a7fc640 10 client.4289.objecter ms_handle_connect 0x7fdd0803e2e0 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 client.?.objecter ms_handle_connect 0x7fe6d0137ce0 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 client.?.objecter ms_handle_connect 0x7fe6d013e5f0 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 824211422 0 0) 0x7fe6c8001410 con 0x7fe6d013e5f0 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): my global_id is 4287 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6bc004ec0 con 0x7fe6d013e5f0 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 client.?.objecter ms_handle_connect 0x7fe6d00a9610 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335801518 0 0) 0x7fe6d00a14b0 con 0x7fe6d0137ce0 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): my global_id is 4295 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6c8001410 con 0x7fe6d0137ce0 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1404673960 0 0) 0x7fe6d013c780 con 0x7fe6d00a9610 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): my global_id is 4291 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6d00a14b0 con 0x7fe6d00a9610 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 811887542 0 0) 0x7fe6bc004ec0 con 0x7fe6d013e5f0 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6d013c780 con 0x7fe6d013e5f0 2024-09-15T20:31:49.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe6c8001ce0 con 0x7fe6d013e5f0 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): dump: 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe6ccff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2364237566 0 0) 0x7fe6c8001410 con 0x7fe6d0137ce0 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe6bc004ec0 con 0x7fe6d0137ce0 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe6c4003ef0 con 0x7fe6d0137ce0 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient(hunting): dump: 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.385 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.386 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.386 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.386 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f71a0003ec0 con 0x7f71ac09d490 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2278634180 0 0) 0x7fe6d013c780 con 0x7fe6d013e5f0 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 >> v1:172.21.15.78:6790/0 conn(0x7fe6d0137ce0 legacy=0x7fe6d01d1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 >> v1:172.21.15.146:6789/0 conn(0x7fe6d00a9610 legacy=0x7fe6d013dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 monclient: found mon.a 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d013c4a0 con 0x7fe6d013e5f0 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 5 monclient: authenticate success, global_id 4287 2024-09-15T20:31:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 10 monclient: _renew_subs 2024-09-15T20:31:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d013e2c0 con 0x7fe6d013e5f0 2024-09-15T20:31:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f71a004b280 con 0x7f71ac09d490 2024-09-15T20:31:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 10 client.4292.objecter ms_dispatch 0x7f71ac000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 3 client.4292.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 3 client.4292.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 10 monclient: _renew_subs 2024-09-15T20:31:49.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6d01cc7b0 con 0x7fe6d013e5f0 2024-09-15T20:31:49.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 1 librados: init done 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 10 client.4292.objecter checking command 1 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 20 client.4292.objecter _get_session s=0x7f71980443d0 osd=1 3 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 20 client.4292.objecter put_session s=0x7f71980443d0 osd=1 3 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 15 client.4292.objecter _session_command_op_remove -1 1 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 20 client.4292.objecter _get_session s=0x7f71980443d0 osd=1 3 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 20 client.4292.objecter get_session s=0x7f71980443d0 osd=1 3 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 15 client.4292.objecter _session_command_op_assign 1 1 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 20 client.4292.objecter put_session s=0x7f71980443d0 osd=1 4 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 10 client.4292.objecter _send_command 1 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 --> v1:172.21.15.78:6800/4006351464 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7198048730 con 0x7f7198044a80 2024-09-15T20:31:49.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 20 client.4292.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7f71a004b650 con 0x7f71ac09d490 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71b0ccc640 10 monclient: get_auth_request con 0x7f7198044a80 auth_method 0 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71ab7fe640 10 monclient: get_auth_request con 0x7f719803df20 auth_method 0 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 10 client.4292.objecter ms_handle_connect 0x7f7198044a80 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe6c8003980 con 0x7fe6d013e5f0 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe6c8003e10 con 0x7fe6d013e5f0 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: dump: 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.311+0000 7fe2f4100640 10 monclient: monmap: 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 librados: starting msgr at 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 librados: starting objecter 2024-09-15T20:31:49.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 Processor -- start 2024-09-15T20:31:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 10 client.4287.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 15 client.4287.objecter _session_command_op_assign -1 1 2024-09-15T20:31:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 20 client.4287.objecter _get_session osd=-1 returning homeless 2024-09-15T20:31:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 10 client.4287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe6d006a080 con 0x7fe6d013e5f0 2024-09-15T20:31:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== osd.4 v1:172.21.15.146:6808/2960966471 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdd08048ad0 con 0x7fdd08044e40 2024-09-15T20:31:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fdd1a7fc640 10 client.4289.objecter ms_dispatch 0x7fdd24000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:31:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fdd1a7fc640 10 client.4289.objecter _finish_command 1 = system:0 2024-09-15T20:31:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fdd1a7fc640 20 client.4289.objecter put_session s=0x7fdd08044790 osd=4 4 2024-09-15T20:31:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fdd1a7fc640 15 client.4289.objecter _session_command_op_remove 4 1 2024-09-15T20:31:49.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6cdffb640 4 set_mon_vals no callback set 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fe6c8004070 con 0x7fe6d013e5f0 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7fe6c804a270 con 0x7fe6d013e5f0 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 client.4287.objecter ms_dispatch 0x7fe6d0000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 3 client.4287.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 3 client.4287.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 client.4287.objecter checking command 1 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 client.4287.objecter _get_session s=0x7fe6bc044880 osd=2 3 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 client.4287.objecter put_session s=0x7fe6bc044880 osd=2 3 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 15 client.4287.objecter _session_command_op_remove -1 1 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 client.4287.objecter _get_session s=0x7fe6bc044880 osd=2 3 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 client.4287.objecter get_session s=0x7fe6bc044880 osd=2 3 2024-09-15T20:31:49.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 15 client.4287.objecter _session_command_op_assign 2 1 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 client.4287.objecter put_session s=0x7fe6bc044880 osd=2 4 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 client.4287.objecter _send_command 1 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6808/404661851 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe6bc048be0 con 0x7fe6bc044f30 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 client.4287.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_get_version_reply(handle=1 version=19) ==== 24+0+0 (unknown 4138596325 0 0) 0x7fe6c804a670 con 0x7fe6d013e5f0 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 monclient: handle_get_version_reply finishing 1 version 19 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== osd.1 v1:172.21.15.78:6800/4006351464 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f7198048730 con 0x7f7198044a80 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 10 client.4292.objecter ms_dispatch 0x7f71ac000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:31:49.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 10 client.4292.objecter _finish_command 1 = system:0 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 20 client.4292.objecter put_session s=0x7f71980443d0 osd=1 4 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7f71a8ff9640 15 client.4292.objecter _session_command_op_remove 1 1 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6cf7fe640 10 monclient: get_auth_request con 0x7fe6bc044f30 auth_method 0 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6cffff640 10 monclient: get_auth_request con 0x7fe6bc03e380 auth_method 0 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 client.4287.objecter ms_handle_connect 0x7fe6bc044f30 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 client.4287.objecter ms_handle_connect 0x7fe6bc03e380 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== osd.2 v1:172.21.15.78:6808/404661851 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe6bc048be0 con 0x7fe6bc044f30 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 client.4287.objecter ms_dispatch 0x7fe6d0000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 10 client.4287.objecter _finish_command 1 = system:0 2024-09-15T20:31:49.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 20 client.4287.objecter put_session s=0x7fe6bc044880 osd=2 4 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe6ccff9640 15 client.4287.objecter _session_command_op_remove 2 1 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 -- start start 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 librados: setting wanted keys 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 librados: calling monclient init 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 10 monclient: init 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 10 monclient(hunting): picked mon.noname-c con 0x7fe2ec0a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 10 monclient(hunting): picked mon.noname-b con 0x7fe2ec137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 10 monclient(hunting): picked mon.noname-a con 0x7fe2ec13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec0a0330 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec0a0610 con 0x7fe2ec137ce0 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2ec13e260 con 0x7fe2ec0a9610 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f4100640 10 monclient(hunting): authenticate will time out at 1170.201777s 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f269d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe2ec13e5f0 0x7fe2ec1d9020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38084/0 (socket says 172.21.15.78:38084) 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f269d640 1 -- 172.21.15.78:0/2291669494 learned_addr learned my addr 172.21.15.78:0/2291669494 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2f1e9c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe2ec0a9610 0x7fe2ec09f0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39344/0 (socket says 172.21.15.78:39344) 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 client.?.objecter ms_handle_connect 0x7fe2ec0a9610 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 client.?.objecter ms_handle_connect 0x7fe2ec13e5f0 2024-09-15T20:31:49.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 client.?.objecter ms_handle_connect 0x7fe2ec137ce0 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2127638080 0 0) 0x7fe2ec13e260 con 0x7fe2ec0a9610 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): my global_id is 4298 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2d0004ef0 con 0x7fe2ec0a9610 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3578446444 0 0) 0x7fe2ec0a0330 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): my global_id is 4290 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec13e260 con 0x7fe2ec13e5f0 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 780317877 0 0) 0x7fe2ec0a0610 con 0x7fe2ec137ce0 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): my global_id is 4294 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe2ec0a0330 con 0x7fe2ec137ce0 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1493612574 0 0) 0x7fe2d0004ef0 con 0x7fe2ec0a9610 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe2ec0a0610 con 0x7fe2ec0a9610 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe2e8001a00 con 0x7fe2ec0a9610 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:49.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): dump: 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2675232619 0 0) 0x7fe2ec0a0610 con 0x7fe2ec0a9610 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 >> v1:172.21.15.146:6789/0 conn(0x7fe2ec137ce0 legacy=0x7fe2ec09f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 >> v1:172.21.15.78:6789/0 conn(0x7fe2ec13e5f0 legacy=0x7fe2ec1d9020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 monclient: found mon.c 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2ec0a14b0 con 0x7fe2ec0a9610 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe2e8003620 con 0x7fe2ec0a9610 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe2e80039b0 con 0x7fe2ec0a9610 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e27fc640 10 monclient: dump: 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.315+0000 7fe2e37fe640 4 set_mon_vals no callback set 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 5 monclient: authenticate success, global_id 4298 2024-09-15T20:31:49.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 10 monclient: _renew_subs 2024-09-15T20:31:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe2ec09ec70 con 0x7fe2ec0a9610 2024-09-15T20:31:49.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7f71a8ff9640 10 client.4292.objecter ms_handle_connect 0x7f719803df20 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 10 monclient: _renew_subs 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe2ec137690 con 0x7fe2ec0a9610 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f4100640 1 librados: init done 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fe2e8003b50 con 0x7fe2ec0a9610 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7fe2e804b2d0 con 0x7fe2ec0a9610 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2e27fc640 10 client.4298.objecter ms_dispatch 0x7fe2ec000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2e27fc640 3 client.4298.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2e27fc640 3 client.4298.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2e27fc640 20 client.4298.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2f169b640 10 monclient: get_auth_request con 0x7fe2d003e3a0 auth_method 0 2024-09-15T20:31:49.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.319+0000 7fe2e27fc640 10 client.4298.objecter ms_handle_connect 0x7fe2d003e3a0 2024-09-15T20:31:49.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 10 client.4298.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 15 client.4298.objecter _session_command_op_assign -1 1 2024-09-15T20:31:49.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 20 client.4298.objecter _get_session s=0x7fe2ec0a0a80 osd=3 3 2024-09-15T20:31:49.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 20 client.4298.objecter put_session s=0x7fe2ec0a0a80 osd=3 3 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 20 client.4298.objecter _get_session s=0x7fe2ec0a0a80 osd=3 3 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 15 client.4298.objecter _session_command_op_remove -1 1 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 20 client.4298.objecter get_session s=0x7fe2ec0a0a80 osd=3 3 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 15 client.4298.objecter _session_command_op_assign 3 1 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 20 client.4298.objecter put_session s=0x7fe2ec0a0a80 osd=3 4 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 10 client.4298.objecter _send_command 1 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.146:6800/490126554 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe2ec06a080 con 0x7fe2ec13eee0 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2f269d640 10 monclient: get_auth_request con 0x7fe2ec13eee0 auth_method 0 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.323+0000 7fe2e27fc640 10 client.4298.objecter ms_handle_connect 0x7fe2ec13eee0 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== osd.3 v1:172.21.15.146:6800/490126554 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe2ec06a080 con 0x7fe2ec13eee0 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fe2e27fc640 10 client.4298.objecter ms_dispatch 0x7fe2ec000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fe2e27fc640 10 client.4298.objecter _finish_command 1 = system:0 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fe2e27fc640 20 client.4298.objecter put_session s=0x7fe2ec0a0a80 osd=3 4 2024-09-15T20:31:49.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fe2e27fc640 15 client.4298.objecter _session_command_op_remove 3 1 2024-09-15T20:31:49.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 10 client.4289.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 15 client.4289.objecter _session_command_op_assign -1 2 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 20 client.4289.objecter _get_session s=0x7fdd08044790 osd=4 3 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 20 client.4289.objecter put_session s=0x7fdd08044790 osd=4 3 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 20 client.4289.objecter _get_session s=0x7fdd08044790 osd=4 3 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 15 client.4289.objecter _session_command_op_remove -1 2 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 20 client.4289.objecter get_session s=0x7fdd08044790 osd=4 3 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 15 client.4289.objecter _session_command_op_assign 4 2 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 20 client.4289.objecter put_session s=0x7fdd08044790 osd=4 4 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 10 client.4289.objecter _send_command 2 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.327+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 --> v1:172.21.15.146:6808/2960966471 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fdd0c001230 con 0x7fdd08044e40 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd1a7fc640 1 -- 172.21.15.78:0/825771535 <== osd.4 v1:172.21.15.146:6808/2960966471 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 559749958) 0x7fdd0c001230 con 0x7fdd08044e40 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd1a7fc640 10 client.4289.objecter ms_dispatch 0x7fdd24000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:31:49.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd1a7fc640 10 client.4289.objecter _finish_command 2 = system:0 2024-09-15T20:31:49.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd1a7fc640 20 client.4289.objecter put_session s=0x7fdd08044790 osd=4 4 2024-09-15T20:31:49.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd1a7fc640 15 client.4289.objecter _session_command_op_remove 4 2 2024-09-15T20:31:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 10 librados: watch_flush enter 2024-09-15T20:31:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 10 librados: watch_flush exit 2024-09-15T20:31:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 10 client.4289.objecter close_session for osd.4 2024-09-15T20:31:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 >> v1:172.21.15.146:6808/2960966471 conn(0x7fdd08044e40 legacy=0x7fdd08047260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 20 client.4289.objecter put_session s=0x7fdd08044790 osd=4 1 2024-09-15T20:31:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 20 client.4289.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 10 client.4289.objecter successfully canceled tick 2024-09-15T20:31:49.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 >> v1:172.21.15.146:6812/435598642 conn(0x7fdd0803e2e0 legacy=0x7fdd080407a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 10 monclient: shutdown 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 >> v1:172.21.15.78:6790/0 conn(0x7fdd24137ce0 legacy=0x7fdd241395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2a17f640 1 -- 172.21.15.78:0/825771535 reap_dead start 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 shutdown_connections 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 >> 172.21.15.78:0/825771535 conn(0x7fdd240aa6c0 msgr2=0x7fdd240a2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 shutdown_connections 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 1 -- 172.21.15.78:0/825771535 wait complete. 2024-09-15T20:31:49.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fdd2bbe2640 1 librados: shutdown 2024-09-15T20:31:49.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 10 client.4287.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T20:31:49.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 15 client.4287.objecter _session_command_op_assign -1 2 2024-09-15T20:31:49.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 20 client.4287.objecter _get_session s=0x7fe6bc044880 osd=2 3 2024-09-15T20:31:49.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 20 client.4287.objecter put_session s=0x7fe6bc044880 osd=2 3 2024-09-15T20:31:49.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 20 client.4287.objecter _get_session s=0x7fe6bc044880 osd=2 3 2024-09-15T20:31:49.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 15 client.4287.objecter _session_command_op_remove -1 2 2024-09-15T20:31:49.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 20 client.4287.objecter get_session s=0x7fe6bc044880 osd=2 3 2024-09-15T20:31:49.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 15 client.4287.objecter _session_command_op_assign 2 2 2024-09-15T20:31:49.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 20 client.4287.objecter put_session s=0x7fe6bc044880 osd=2 4 2024-09-15T20:31:49.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 10 client.4287.objecter _send_command 2 2024-09-15T20:31:49.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 --> v1:172.21.15.78:6808/404661851 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe6d00a8f00 con 0x7fe6bc044f30 2024-09-15T20:31:49.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6ccff9640 1 -- 172.21.15.78:0/3340253074 <== osd.2 v1:172.21.15.78:6808/404661851 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3543570501) 0x7fe6d00a8f00 con 0x7fe6bc044f30 2024-09-15T20:31:49.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6ccff9640 10 client.4287.objecter ms_dispatch 0x7fe6d0000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:31:49.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6ccff9640 10 client.4287.objecter _finish_command 2 = system:0 2024-09-15T20:31:49.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6ccff9640 20 client.4287.objecter put_session s=0x7fe6bc044880 osd=2 4 2024-09-15T20:31:49.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6ccff9640 15 client.4287.objecter _session_command_op_remove 2 2 2024-09-15T20:31:49.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 10 librados: watch_flush enter 2024-09-15T20:31:49.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 10 librados: watch_flush exit 2024-09-15T20:31:49.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 10 client.4287.objecter close_session for osd.2 2024-09-15T20:31:49.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 >> v1:172.21.15.78:6808/404661851 conn(0x7fe6bc044f30 legacy=0x7fe6bc047370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 20 client.4287.objecter put_session s=0x7fe6bc044880 osd=2 1 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 20 client.4287.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.331+0000 7fe6d68df640 10 client.4287.objecter successfully canceled tick 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 >> v1:172.21.15.146:6812/435598642 conn(0x7fe6bc03e380 legacy=0x7fe6bc040840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 10 monclient: shutdown 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 >> v1:172.21.15.78:6789/0 conn(0x7fe6d013e5f0 legacy=0x7fe6d01d5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d4e7c640 1 -- 172.21.15.78:0/3340253074 reap_dead start 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 shutdown_connections 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 >> 172.21.15.78:0/3340253074 conn(0x7fe6d00aa6c0 msgr2=0x7fe6d0073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 shutdown_connections 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 1 -- 172.21.15.78:0/3340253074 wait complete. 2024-09-15T20:31:49.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7fe6d68df640 1 librados: shutdown 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 10 client.4292.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 15 client.4292.objecter _session_command_op_assign -1 2 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 20 client.4292.objecter _get_session s=0x7f71980443d0 osd=1 3 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 20 client.4292.objecter put_session s=0x7f71980443d0 osd=1 3 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 20 client.4292.objecter _get_session s=0x7f71980443d0 osd=1 3 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 15 client.4292.objecter _session_command_op_remove -1 2 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 20 client.4292.objecter get_session s=0x7f71980443d0 osd=1 3 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 15 client.4292.objecter _session_command_op_assign 1 2 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 20 client.4292.objecter put_session s=0x7f71980443d0 osd=1 4 2024-09-15T20:31:49.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 10 client.4292.objecter _send_command 2 2024-09-15T20:31:49.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 --> v1:172.21.15.78:6800/4006351464 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f71ac02e5d0 con 0x7f7198044a80 2024-09-15T20:31:49.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71a8ff9640 1 -- 172.21.15.78:0/1792947077 <== osd.1 v1:172.21.15.78:6800/4006351464 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7f71ac02e5d0 con 0x7f7198044a80 2024-09-15T20:31:49.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71a8ff9640 10 client.4292.objecter ms_dispatch 0x7f71ac000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:31:49.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71a8ff9640 10 client.4292.objecter _finish_command 2 = system:0 2024-09-15T20:31:49.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71a8ff9640 20 client.4292.objecter put_session s=0x7f71980443d0 osd=1 4 2024-09-15T20:31:49.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71a8ff9640 15 client.4292.objecter _session_command_op_remove 1 2 2024-09-15T20:31:49.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 10 librados: watch_flush enter 2024-09-15T20:31:49.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 10 librados: watch_flush exit 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 10 client.4292.objecter close_session for osd.1 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 >> v1:172.21.15.78:6800/4006351464 conn(0x7f7198044a80 legacy=0x7f7198046ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 20 client.4292.objecter put_session s=0x7f71980443d0 osd=1 1 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 20 client.4292.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 10 client.4292.objecter successfully canceled tick 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 >> v1:172.21.15.146:6812/435598642 conn(0x7f719803df20 legacy=0x7f71980403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 10 monclient: shutdown 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 >> v1:172.21.15.78:6790/0 conn(0x7f71ac09d490 legacy=0x7f71ac1d1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b0ccc640 1 -- 172.21.15.78:0/1792947077 reap_dead start 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 shutdown_connections 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 >> 172.21.15.78:0/1792947077 conn(0x7f71ac0a2a90 msgr2=0x7f71ac0741d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 shutdown_connections 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 -- 172.21.15.78:0/1792947077 wait complete. 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.335+0000 7f71b272f640 1 librados: shutdown 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 10 client.4298.objecter _submit_command 2 [{"prefix": "flush_pg_stats"}] 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 15 client.4298.objecter _session_command_op_assign -1 2 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 20 client.4298.objecter _get_session s=0x7fe2ec0a0a80 osd=3 3 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 20 client.4298.objecter put_session s=0x7fe2ec0a0a80 osd=3 3 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 20 client.4298.objecter _get_session s=0x7fe2ec0a0a80 osd=3 3 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 15 client.4298.objecter _session_command_op_remove -1 2 2024-09-15T20:31:49.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 20 client.4298.objecter get_session s=0x7fe2ec0a0a80 osd=3 3 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 15 client.4298.objecter _session_command_op_assign 3 2 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 20 client.4298.objecter put_session s=0x7fe2ec0a0a80 osd=3 4 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 10 client.4298.objecter _send_command 2 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.351+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 --> v1:172.21.15.146:6800/490126554 -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe2ec02e460 con 0x7fe2ec13eee0 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2e27fc640 1 -- 172.21.15.78:0/2291669494 <== osd.3 v1:172.21.15.146:6800/490126554 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (unknown 0 0 3580782990) 0x7fe2ec02e460 con 0x7fe2ec13eee0 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2e27fc640 10 client.4298.objecter ms_dispatch 0x7fe2ec000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2e27fc640 10 client.4298.objecter _finish_command 2 = system:0 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2e27fc640 20 client.4298.objecter put_session s=0x7fe2ec0a0a80 osd=3 4 2024-09-15T20:31:49.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2e27fc640 15 client.4298.objecter _session_command_op_remove 3 2 2024-09-15T20:31:49.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 10 librados: watch_flush enter 2024-09-15T20:31:49.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 10 librados: watch_flush exit 2024-09-15T20:31:49.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 10 client.4298.objecter close_session for osd.3 2024-09-15T20:31:49.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 >> v1:172.21.15.146:6800/490126554 conn(0x7fe2ec13eee0 legacy=0x7fe2ec13f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 20 client.4298.objecter put_session s=0x7fe2ec0a0a80 osd=3 1 2024-09-15T20:31:49.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 20 client.4298.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 10 client.4298.objecter successfully canceled tick 2024-09-15T20:31:49.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 >> v1:172.21.15.146:6812/435598642 conn(0x7fe2d003e3a0 legacy=0x7fe2d0040860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 10 monclient: shutdown 2024-09-15T20:31:49.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 >> v1:172.21.15.78:6790/0 conn(0x7fe2ec0a9610 legacy=0x7fe2ec09f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f269d640 1 -- 172.21.15.78:0/2291669494 reap_dead start 2024-09-15T20:31:49.426 INFO:teuthology.orchestra.run.smithi078.stdout:51539607557 2024-09-15T20:31:49.427 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:31:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 shutdown_connections 2024-09-15T20:31:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 >> 172.21.15.78:0/2291669494 conn(0x7fe2ec0aa6c0 msgr2=0x7fe2ec0a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 shutdown_connections 2024-09-15T20:31:49.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 1 -- 172.21.15.78:0/2291669494 wait complete. 2024-09-15T20:31:49.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.355+0000 7fe2f4100640 1 librados: shutdown 2024-09-15T20:31:49.428 INFO:teuthology.orchestra.run.smithi078.stdout:51539607556 2024-09-15T20:31:49.428 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T20:31:49.429 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-09-15T20:31:49.429 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:31:49.430 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-09-15T20:31:49.430 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:31:49.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient: monmap: 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.445171+0000 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.445171+0000 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.446 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient: monmap: 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.445209+0000 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.445209+0000 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.447 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.448 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.448 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 Processor -- start 2024-09-15T20:31:49.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- start start 2024-09-15T20:31:49.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient: init 2024-09-15T20:31:49.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 Processor -- start 2024-09-15T20:31:49.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- start start 2024-09-15T20:31:49.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient: init 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient(hunting): picked mon.noname-b con 0x7fb228137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient(hunting): picked mon.noname-c con 0x7fb2280a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient(hunting): picked mon.noname-a con 0x7fb22813e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2280a6420 con 0x7fb22813e5f0 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2280a6700 con 0x7fb2280a9610 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2280a2110 con 0x7fb228137ce0 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient(hunting): authenticate will time out at 1170.329329s 2024-09-15T20:31:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22e7bc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb22813e5f0 0x7fb2281409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38098/0 (socket says 172.21.15.78:38098) 2024-09-15T20:31:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22dfbb640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb228137ce0 0x7fb2281380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39360/0 (socket says 172.21.15.78:39360) 2024-09-15T20:31:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22e7bc640 1 -- 172.21.15.78:0/3555208865 learned_addr learned my addr 172.21.15.78:0/3555208865 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient(hunting): picked mon.noname-c con 0x7ff9100a3ea0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient(hunting): picked mon.noname-b con 0x7ff91009cee0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90e7fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff91009cee0 0x7ff91013aa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39374/0 (socket says 172.21.15.78:39374) 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient(hunting): picked mon.noname-a con 0x7ff9100a6360 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9100a07b0 con 0x7ff9100a6360 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff9100a6360 0x7ff91013e180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38108/0 (socket says 172.21.15.78:38108) 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90f7fe640 1 -- 172.21.15.78:0/1361360846 learned_addr learned my addr 172.21.15.78:0/1361360846 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- 172.21.15.78:0/1361360846 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9100a0a90 con 0x7ff9100a3ea0 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- 172.21.15.78:0/1361360846 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff91009cc60 con 0x7ff91009cee0 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient(hunting): authenticate will time out at 1170.329849s 2024-09-15T20:31:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4237815022 0 0) 0x7fb2280a6420 con 0x7fb22813e5f0 2024-09-15T20:31:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient(hunting): my global_id is 4293 2024-09-15T20:31:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb214003530 con 0x7fb22813e5f0 2024-09-15T20:31:49.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 222209701 0 0) 0x7fb2280a2110 con 0x7fb228137ce0 2024-09-15T20:31:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient(hunting): my global_id is 4301 2024-09-15T20:31:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2280a6420 con 0x7fb228137ce0 2024-09-15T20:31:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2801913400 0 0) 0x7fb214003530 con 0x7fb22813e5f0 2024-09-15T20:31:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 >> v1:172.21.15.78:6790/0 conn(0x7fb228137ce0 legacy=0x7fb2281380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 >> v1:172.21.15.146:6789/0 conn(0x7fb2280a9610 legacy=0x7fb22813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 monclient: found mon.noname-a 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2280a4bc0 con 0x7fb22813e5f0 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb220001a20 con 0x7fb22813e5f0 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 5 monclient: authenticate success, global_id 4293 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1129284395 0 0) 0x7ff9100a07b0 con 0x7ff9100a6360 2024-09-15T20:31:49.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient(hunting): my global_id is 4296 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: dump: 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.457 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.458 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.458 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.458 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.458 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.458 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8f8003530 con 0x7ff9100a6360 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16300057 0 0) 0x7ff91009cc60 con 0x7ff91009cee0 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient(hunting): my global_id is 4304 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9100a07b0 con 0x7ff91009cee0 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2327084265 0 0) 0x7ff8f8003530 con 0x7ff9100a6360 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 >> v1:172.21.15.78:6790/0 conn(0x7ff91009cee0 legacy=0x7ff91013aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 >> v1:172.21.15.146:6789/0 conn(0x7ff9100a3ea0 legacy=0x7ff9100a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 monclient: found mon.noname-a 2024-09-15T20:31:49.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9100773b0 con 0x7ff9100a6360 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff908001a20 con 0x7ff9100a6360 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: dump: 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.460 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb220001bc0 con 0x7fb22813e5f0 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 1 -- 172.21.15.78:0/3555208865 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb220001fd0 con 0x7fb22813e5f0 2024-09-15T20:31:49.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb22cfb9640 10 monclient: dump: 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 4 set_mon_vals no callback set 2024-09-15T20:31:49.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 5 monclient: authenticate success, global_id 4296 2024-09-15T20:31:49.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff908001bc0 con 0x7ff9100a6360 2024-09-15T20:31:49.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 1 -- 172.21.15.78:0/1361360846 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff908002010 con 0x7ff9100a6360 2024-09-15T20:31:49.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff90dffb640 10 monclient: dump: 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 4 set_mon_vals no callback set 2024-09-15T20:31:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 10 monclient: shutdown 2024-09-15T20:31:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- 172.21.15.78:0/3555208865 >> v1:172.21.15.78:6789/0 conn(0x7fb22813e5f0 legacy=0x7fb2281409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- 172.21.15.78:0/3555208865 shutdown_connections 2024-09-15T20:31:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- 172.21.15.78:0/3555208865 >> 172.21.15.78:0/3555208865 conn(0x7fb2280aa6c0 msgr2=0x7fb2280aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 10 monclient: shutdown 2024-09-15T20:31:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- 172.21.15.78:0/1361360846 >> v1:172.21.15.78:6789/0 conn(0x7ff9100a6360 legacy=0x7ff91013e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- 172.21.15.78:0/3555208865 shutdown_connections 2024-09-15T20:31:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- 172.21.15.78:0/1361360846 shutdown_connections 2024-09-15T20:31:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- 172.21.15.78:0/1361360846 >> 172.21.15.78:0/1361360846 conn(0x7ff9100a3380 msgr2=0x7ff9100a3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- 172.21.15.78:0/1361360846 shutdown_connections 2024-09-15T20:31:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7ff9157d8640 1 -- 172.21.15.78:0/1361360846 wait complete. 2024-09-15T20:31:49.467 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.443+0000 7fb23021f640 1 -- 172.21.15.78:0/3555208865 wait complete. 2024-09-15T20:31:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient: monmap: 2024-09-15T20:31:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.468 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.469 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 librados: starting msgr at 2024-09-15T20:31:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 librados: starting objecter 2024-09-15T20:31:49.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 Processor -- start 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: monmap: 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.470 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 librados: starting msgr at 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 librados: starting objecter 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 Processor -- start 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 -- start start 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 librados: setting wanted keys 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 librados: calling monclient init 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: init 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient(hunting): picked mon.noname-c con 0x7fb2280a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient(hunting): picked mon.noname-a con 0x7fb228137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient(hunting): picked mon.noname-b con 0x7fb22813e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22813c070 con 0x7fb228137ce0 2024-09-15T20:31:49.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2280a14b0 con 0x7fb22813e5f0 2024-09-15T20:31:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22813a870 con 0x7fb2280a9610 2024-09-15T20:31:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient(hunting): authenticate will time out at 1170.333209s 2024-09-15T20:31:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 -- start start 2024-09-15T20:31:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 librados: setting wanted keys 2024-09-15T20:31:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 librados: calling monclient init 2024-09-15T20:31:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient: init 2024-09-15T20:31:49.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb22dfbb640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb2280a9610 0x7fb2281ccac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39380/0 (socket says 172.21.15.78:39380) 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb22dfbb640 1 -- 172.21.15.78:0/111085953 learned_addr learned my addr 172.21.15.78:0/111085953 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 client.?.objecter ms_handle_connect 0x7fb228137ce0 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 client.?.objecter ms_handle_connect 0x7fb2280a9610 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient(hunting): picked mon.noname-a con 0x7ff9100a3ea0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient(hunting): picked mon.noname-c con 0x7ff9100a6360 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient(hunting): picked mon.noname-b con 0x7ff91009eb40 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff910136630 con 0x7ff9100a3ea0 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff90effd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff9100a3ea0 0x7ff9101d1060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38130/0 (socket says 172.21.15.78:38130) 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff90effd640 1 -- 172.21.15.78:0/2382589711 learned_addr learned my addr 172.21.15.78:0/2382589711 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff910136910 con 0x7ff91009eb40 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff9101353f0 con 0x7ff9100a6360 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff9157d8640 10 monclient(hunting): authenticate will time out at 1170.334146s 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1023752340 0 0) 0x7fb22813c070 con 0x7fb228137ce0 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): my global_id is 4299 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb20c004ec0 con 0x7fb228137ce0 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4212838082 0 0) 0x7fb22813a870 con 0x7fb2280a9610 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): my global_id is 4307 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb22813c070 con 0x7fb2280a9610 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 client.?.objecter ms_handle_connect 0x7ff9100a3ea0 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 client.?.objecter ms_handle_connect 0x7fb22813e5f0 2024-09-15T20:31:49.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 911933008 0 0) 0x7fb20c004ec0 con 0x7fb228137ce0 2024-09-15T20:31:49.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 client.?.objecter ms_handle_connect 0x7ff9100a6360 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2524203275 0 0) 0x7ff910136630 con 0x7ff9100a3ea0 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): my global_id is 4302 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8fc004ec0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 client.?.objecter ms_handle_connect 0x7ff91009eb40 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb22813a870 con 0x7fb228137ce0 2024-09-15T20:31:49.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1797051133 0 0) 0x7fb2280a14b0 con 0x7fb22813e5f0 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): my global_id is 4303 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb20c004ec0 con 0x7fb22813e5f0 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3523583824 0 0) 0x7fb22813c070 con 0x7fb2280a9610 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2280a14b0 con 0x7fb2280a9610 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2180018a0 con 0x7fb228137ce0 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): dump: 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.478 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2240013c0 con 0x7fb2280a9610 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient(hunting): dump: 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.479 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3625874434 0 0) 0x7fb22813a870 con 0x7fb228137ce0 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 >> v1:172.21.15.78:6790/0 conn(0x7fb2280a9610 legacy=0x7fb2281ccac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 >> v1:172.21.15.146:6789/0 conn(0x7fb22813e5f0 legacy=0x7fb2281d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 monclient: found mon.a 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22813aab0 con 0x7fb228137ce0 2024-09-15T20:31:49.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1413437720 0 0) 0x7ff9101353f0 con 0x7ff9100a6360 2024-09-15T20:31:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): my global_id is 4310 2024-09-15T20:31:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff910136630 con 0x7ff9100a6360 2024-09-15T20:31:49.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793103885 0 0) 0x7ff910136910 con 0x7ff91009eb40 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): my global_id is 4306 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff9101353f0 con 0x7ff91009eb40 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1809334336 0 0) 0x7ff8fc004ec0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff910136910 con 0x7ff9100a3ea0 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff9000019c0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): dump: 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.483 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 609656773 0 0) 0x7ff910136630 con 0x7ff9100a6360 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8fc004ec0 con 0x7ff9100a6360 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff904001820 con 0x7ff9100a6360 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7ff8f77fe640 10 monclient(hunting): dump: 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.484 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.485 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.485 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 5 monclient: authenticate success, global_id 4299 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: _renew_subs 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb2281cd1d0 con 0x7fb228137ce0 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 client.4299.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: _renew_subs 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2280a0100 con 0x7fb228137ce0 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb23021f640 1 librados: init done 2024-09-15T20:31:49.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb218003600 con 0x7fb228137ce0 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2180039d0 con 0x7fb228137ce0 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21e7fc640 10 monclient: dump: 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.447+0000 7fb21ffff640 4 set_mon_vals no callback set 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb23021f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb23021f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb218003b70 con 0x7fb228137ce0 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb23021f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb210005090 con 0x7fb228137ce0 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7fb218049e70 con 0x7fb228137ce0 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb21e7fc640 10 client.4299.objecter ms_dispatch 0x7fb228000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb21e7fc640 3 client.4299.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb21e7fc640 3 client.4299.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb21e7fc640 20 client.4299.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb22dfbb640 10 monclient: get_auth_request con 0x7fb20c03e350 auth_method 0 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7fb21e7fc640 10 client.4299.objecter ms_handle_connect 0x7fb20c03e350 2024-09-15T20:31:49.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 939915558 0 0) 0x7ff910136910 con 0x7ff9100a3ea0 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 >> v1:172.21.15.78:6790/0 conn(0x7ff9100a6360 legacy=0x7ff9101d6250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 >> v1:172.21.15.146:6789/0 conn(0x7ff91009eb40 legacy=0x7ff9101d9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 1 monclient: found mon.a 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff910130b30 con 0x7ff9100a3ea0 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff8f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 5 monclient: authenticate success, global_id 4302 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 10 monclient: _renew_subs 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff9101305e0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 10 client.4302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 10 monclient: _renew_subs 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff9101d1770 con 0x7ff9100a3ea0 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 1 librados: init done 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.451+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff9100008e0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb218015870 con 0x7fb228137ce0 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7fb21e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7fb21e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:49.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff900003620 con 0x7ff9100a3ea0 2024-09-15T20:31:49.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7ff8f77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff900003a70 con 0x7ff9100a3ea0 2024-09-15T20:31:49.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7ff8f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7ff8f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7ff8f77fe640 10 monclient: dump: 2024-09-15T20:31:49.491 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.455+0000 7ff90d7fa640 4 set_mon_vals no callback set 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.459+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff900003a70 con 0x7ff9100a3ea0 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.459+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7ff90004aee0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.459+0000 7ff8f77fe640 10 client.4302.objecter ms_dispatch 0x7ff910000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.459+0000 7ff8f77fe640 3 client.4302.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.459+0000 7ff8f77fe640 3 client.4302.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.459+0000 7ff8f77fe640 20 client.4302.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.459+0000 7ff90e7fc640 10 monclient: get_auth_request con 0x7ff8fc03e040 auth_method 0 2024-09-15T20:31:49.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.459+0000 7ff8f77fe640 10 client.4302.objecter ms_handle_connect 0x7ff8fc03e040 2024-09-15T20:31:49.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.463+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7ff9000168e0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.463+0000 7ff8f77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.463+0000 7ff8f77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient: monmap: 2024-09-15T20:31:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.610477+0000 2024-09-15T20:31:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.610477+0000 2024-09-15T20:31:49.611 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.612 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 1 Processor -- start 2024-09-15T20:31:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 1 -- start start 2024-09-15T20:31:49.612 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient: init 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7feac997c640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7feac997c640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7feac997c640 10 monclient: monmap: 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.611800+0000 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.611800+0000 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.613 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.614 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.614 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.614 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:49.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient(hunting): picked mon.noname-a con 0x7fbf90137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient(hunting): picked mon.noname-c con 0x7fbf900a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf8f7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbf90137ce0 0x7fbf901380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38134/0 (socket says 172.21.15.78:38134) 2024-09-15T20:31:49.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf8ffff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fbf9013e5f0 0x7fbf901409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39392/0 (socket says 172.21.15.78:39392) 2024-09-15T20:31:49.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7feac997c640 1 Processor -- start 2024-09-15T20:31:49.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7feac997c640 1 -- start start 2024-09-15T20:31:49.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7feac997c640 10 monclient: init 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 10 monclient(hunting): picked mon.noname-b con 0x7fbf9013e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf8f7fe640 1 -- 172.21.15.78:0/3742799552 learned_addr learned my addr 172.21.15.78:0/3742799552 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 1 -- 172.21.15.78:0/3742799552 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf900a6420 con 0x7fbf90137ce0 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.607+0000 7fbf961c7640 1 -- 172.21.15.78:0/3742799552 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf900a6700 con 0x7fbf900a9610 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- 172.21.15.78:0/3742799552 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf900a2110 con 0x7fbf9013e5f0 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient(hunting): authenticate will time out at 1170.496064s 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 382087417 0 0) 0x7fbf900a6420 con 0x7fbf90137ce0 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): my global_id is 4305 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf7c003530 con 0x7fbf90137ce0 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1064797069 0 0) 0x7fbf900a2110 con 0x7fbf9013e5f0 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): my global_id is 4313 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf900a6420 con 0x7fbf9013e5f0 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000423643 0 0) 0x7fbf900a6700 con 0x7fbf900a9610 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient(hunting): my global_id is 4309 2024-09-15T20:31:49.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf900a2110 con 0x7fbf900a9610 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient(hunting): picked mon.noname-b con 0x7feac4137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient(hunting): picked mon.noname-a con 0x7feac40a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient(hunting): picked mon.noname-c con 0x7feac413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac40a6420 con 0x7feac40a9610 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac40a6700 con 0x7feac413e5f0 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac40a2110 con 0x7feac4137ce0 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient(hunting): authenticate will time out at 1170.496689s 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7feac4137ce0 0x7feac41380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39400/0 (socket says 172.21.15.78:39400) 2024-09-15T20:31:49.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac2ffd640 1 -- 172.21.15.78:0/1337679732 learned_addr learned my addr 172.21.15.78:0/1337679732 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2999342027 0 0) 0x7fbf7c003530 con 0x7fbf90137ce0 2024-09-15T20:31:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 >> v1:172.21.15.78:6790/0 conn(0x7fbf9013e5f0 legacy=0x7fbf901409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 >> v1:172.21.15.146:6789/0 conn(0x7fbf900a9610 legacy=0x7fbf9013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 monclient: found mon.noname-a 2024-09-15T20:31:49.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf900a4bc0 con 0x7fbf90137ce0 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf800016a0 con 0x7fbf90137ce0 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: dump: 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.620 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 5 monclient: authenticate success, global_id 4305 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf80001840 con 0x7fbf90137ce0 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 1 -- 172.21.15.78:0/3742799552 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf80001c10 con 0x7fbf90137ce0 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf8effd640 10 monclient: dump: 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.621 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 4 set_mon_vals no callback set 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient: shutdown 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- 172.21.15.78:0/3742799552 >> v1:172.21.15.78:6789/0 conn(0x7fbf90137ce0 legacy=0x7fbf901380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- 172.21.15.78:0/3742799552 shutdown_connections 2024-09-15T20:31:49.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- 172.21.15.78:0/3742799552 >> 172.21.15.78:0/3742799552 conn(0x7fbf900aa6c0 msgr2=0x7fbf900aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- 172.21.15.78:0/3742799552 shutdown_connections 2024-09-15T20:31:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- 172.21.15.78:0/3742799552 wait complete. 2024-09-15T20:31:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 58208210 0 0) 0x7feac40a2110 con 0x7feac4137ce0 2024-09-15T20:31:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): my global_id is 4316 2024-09-15T20:31:49.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaac003530 con 0x7feac4137ce0 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1945996862 0 0) 0x7feac40a6420 con 0x7feac40a9610 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): my global_id is 4308 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feac40a2110 con 0x7feac40a9610 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524887330 0 0) 0x7feac40a6700 con 0x7feac413e5f0 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient(hunting): my global_id is 4312 2024-09-15T20:31:49.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feac40a6420 con 0x7feac413e5f0 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7efc4a084640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7efc4a084640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7efc4a084640 10 monclient: monmap: 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.615375+0000 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.615375+0000 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.625 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 718548664 0 0) 0x7feac40a2110 con 0x7feac40a9610 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 >> v1:172.21.15.78:6790/0 conn(0x7feac4137ce0 legacy=0x7feac41380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 >> v1:172.21.15.146:6789/0 conn(0x7feac413e5f0 legacy=0x7feac41409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 monclient: found mon.noname-a 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac40a4bc0 con 0x7feac40a9610 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feab4001530 con 0x7feac40a9610 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: dump: 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.627 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 5 monclient: authenticate success, global_id 4308 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feab40016d0 con 0x7feac40a9610 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 1 -- 172.21.15.78:0/1337679732 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feab4001aa0 con 0x7feac40a9610 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 4 set_mon_vals no callback set 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac1ffb640 10 monclient: dump: 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.628 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient: monmap: 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.629 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 librados: starting msgr at 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 librados: starting objecter 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 Processor -- start 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- start start 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 librados: setting wanted keys 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 librados: calling monclient init 2024-09-15T20:31:49.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient: init 2024-09-15T20:31:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 10 monclient: shutdown 2024-09-15T20:31:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7feac997c640 1 -- 172.21.15.78:0/1337679732 >> v1:172.21.15.78:6789/0 conn(0x7feac40a9610 legacy=0x7feac413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient(hunting): picked mon.noname-c con 0x7fbf900a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient(hunting): picked mon.noname-b con 0x7fbf90137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient(hunting): picked mon.noname-a con 0x7fbf9013e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf90139cd0 con 0x7fbf9013e5f0 2024-09-15T20:31:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf900a14b0 con 0x7fbf90137ce0 2024-09-15T20:31:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf9013b6c0 con 0x7fbf900a9610 2024-09-15T20:31:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7fbf961c7640 10 monclient(hunting): authenticate will time out at 1170.499904s 2024-09-15T20:31:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 -- 172.21.15.78:0/1337679732 shutdown_connections 2024-09-15T20:31:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 -- 172.21.15.78:0/1337679732 >> 172.21.15.78:0/1337679732 conn(0x7feac40aa6c0 msgr2=0x7feac40aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 -- 172.21.15.78:0/1337679732 shutdown_connections 2024-09-15T20:31:49.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7efc4a084640 1 Processor -- start 2024-09-15T20:31:49.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7efc4a084640 1 -- start start 2024-09-15T20:31:49.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.611+0000 7efc4a084640 10 monclient: init 2024-09-15T20:31:49.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 -- 172.21.15.78:0/1337679732 wait complete. 2024-09-15T20:31:49.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8f7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fbf900a9610 0x7fbf90138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39414/0 (socket says 172.21.15.78:39414) 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8f7fe640 1 -- 172.21.15.78:0/238690155 learned_addr learned my addr 172.21.15.78:0/238690155 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 client.?.objecter ms_handle_connect 0x7fbf900a9610 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient: monmap: 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.634 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.635 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 client.?.objecter ms_handle_connect 0x7fbf90137ce0 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2140572245 0 0) 0x7fbf9013b6c0 con 0x7fbf900a9610 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): my global_id is 4319 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf70004ec0 con 0x7fbf900a9610 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 100778193 0 0) 0x7fbf900a14b0 con 0x7fbf90137ce0 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): my global_id is 4315 2024-09-15T20:31:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf9013b6c0 con 0x7fbf90137ce0 2024-09-15T20:31:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 librados: starting msgr at 2024-09-15T20:31:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 librados: starting objecter 2024-09-15T20:31:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3346590323 0 0) 0x7fbf70004ec0 con 0x7fbf900a9610 2024-09-15T20:31:49.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf900a14b0 con 0x7fbf900a9610 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf80001b90 con 0x7fbf900a9610 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): dump: 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.638 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 Processor -- start 2024-09-15T20:31:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 -- start start 2024-09-15T20:31:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 librados: setting wanted keys 2024-09-15T20:31:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 librados: calling monclient init 2024-09-15T20:31:49.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient: init 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2395278177 0 0) 0x7fbf900a14b0 con 0x7fbf900a9610 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 >> v1:172.21.15.146:6789/0 conn(0x7fbf90137ce0 legacy=0x7fbf901395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 >> v1:172.21.15.78:6789/0 conn(0x7fbf9013e5f0 legacy=0x7fbf901d51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 monclient: found mon.c 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9013c530 con 0x7fbf900a9610 2024-09-15T20:31:49.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient(hunting): picked mon.noname-a con 0x7efc4413c250 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient(hunting): picked mon.noname-c con 0x7efc4413cd40 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient(hunting): picked mon.noname-b con 0x7efc440a9500 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc440a2370 con 0x7efc4413c250 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc440a4bc0 con 0x7efc4413cd40 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc440a17c0 con 0x7efc440a9500 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient(hunting): authenticate will time out at 1170.500219s 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc437fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efc4413c250 0x7efc4413c630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38162/0 (socket says 172.21.15.78:38162) 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc437fe640 1 -- 172.21.15.78:0/3408442727 learned_addr learned my addr 172.21.15.78:0/3408442727 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3421509071 0 0) 0x7efc440a2370 con 0x7efc4413c250 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient(hunting): my global_id is 4311 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc2c003530 con 0x7efc4413c250 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1170233778 0 0) 0x7efc2c003530 con 0x7efc4413c250 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 >> v1:172.21.15.78:6790/0 conn(0x7efc440a9500 legacy=0x7efc44144bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 >> v1:172.21.15.146:6789/0 conn(0x7efc4413cd40 legacy=0x7efc44141490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 monclient: found mon.noname-a 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc440a6460 con 0x7efc4413c250 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efc380016a0 con 0x7efc4413c250 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: dump: 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efc38001840 con 0x7efc4413c250 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 1 -- 172.21.15.78:0/3408442727 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efc38001c90 con 0x7efc4413c250 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc427fc640 10 monclient: dump: 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.645 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 5 monclient: authenticate success, global_id 4311 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 4 set_mon_vals no callback set 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7f9d71488640 10 monclient: get_monmap_and_config 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7f9d71488640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7f9d71488640 10 monclient: monmap: 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.646 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:49.619203+0000 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:49.619203+0000 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:49.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient(hunting): picked mon.noname-c con 0x7feac40a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient(hunting): picked mon.noname-b con 0x7feac4137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient(hunting): picked mon.noname-a con 0x7feac413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac4139cd0 con 0x7feac413e5f0 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac4074450 con 0x7feac4137ce0 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feac40a14b0 con 0x7feac40a9610 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac997c640 10 monclient(hunting): authenticate will time out at 1170.502271s 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7feac40a9610 0x7feac4138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:39436/0 (socket says 172.21.15.78:39436) 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feac2ffd640 1 -- 172.21.15.78:0/2964418763 learned_addr learned my addr 172.21.15.78:0/2964418763 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 client.?.objecter ms_handle_connect 0x7feac40a9610 2024-09-15T20:31:49.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 5 monclient: authenticate success, global_id 4319 2024-09-15T20:31:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 10 monclient: _renew_subs 2024-09-15T20:31:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9013ca30 con 0x7fbf900a9610 2024-09-15T20:31:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 10 monclient: _renew_subs 2024-09-15T20:31:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf9013dd90 con 0x7fbf900a9610 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 1 librados: init done 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf80001820 con 0x7fbf900a9610 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf80004da0 con 0x7fbf900a9610 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8e7fc640 4 set_mon_vals no callback set 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 monclient: dump: 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.650 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 10 monclient: shutdown 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 1 -- 172.21.15.78:0/3408442727 >> v1:172.21.15.78:6789/0 conn(0x7efc4413c250 legacy=0x7efc4413c630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 1 -- 172.21.15.78:0/3408442727 shutdown_connections 2024-09-15T20:31:49.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 1 -- 172.21.15.78:0/3408442727 >> 172.21.15.78:0/3408442727 conn(0x7efc440aa6c0 msgr2=0x7efc440aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7f9d71488640 1 Processor -- start 2024-09-15T20:31:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 1 -- 172.21.15.78:0/3408442727 shutdown_connections 2024-09-15T20:31:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fbf80010e10 con 0x7fbf900a9610 2024-09-15T20:31:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7fbf8004b090 con 0x7fbf900a9610 2024-09-15T20:31:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 10 client.4319.objecter ms_dispatch 0x7fbf90000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 3 client.4319.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 3 client.4319.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf8cff9640 20 client.4319.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 client.?.objecter ms_handle_connect 0x7feac4137ce0 2024-09-15T20:31:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 client.?.objecter ms_handle_connect 0x7feac413e5f0 2024-09-15T20:31:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 512608154 0 0) 0x7feac4074450 con 0x7feac4137ce0 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): my global_id is 4321 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feab0004ec0 con 0x7feac4137ce0 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700183714 0 0) 0x7feac4139cd0 con 0x7feac413e5f0 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): my global_id is 4314 2024-09-15T20:31:49.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feac4074450 con 0x7feac413e5f0 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 740794405 0 0) 0x7feac4074450 con 0x7feac413e5f0 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feac4139cd0 con 0x7feac413e5f0 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feabc001a30 con 0x7feac413e5f0 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): dump: 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.655 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2970860445 0 0) 0x7feab0004ec0 con 0x7feac4137ce0 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feac4074450 con 0x7feac4137ce0 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1214027955 0 0) 0x7feac40a14b0 con 0x7feac40a9610 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient(hunting): my global_id is 4322 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feab0004ec0 con 0x7feac40a9610 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7f9d71488640 1 -- start start 2024-09-15T20:31:49.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7f9d71488640 10 monclient: init 2024-09-15T20:31:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf74005090 con 0x7fbf900a9610 2024-09-15T20:31:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7fbf87fff640 10 monclient: get_auth_request con 0x7fbf7003e0d0 auth_method 0 2024-09-15T20:31:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7fbf8cff9640 10 client.4319.objecter ms_handle_connect 0x7fbf7003e0d0 2024-09-15T20:31:49.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 10 monclient(hunting): picked mon.noname-c con 0x7f9d6c09b3d0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 10 monclient(hunting): picked mon.noname-a con 0x7f9d6c0a2410 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6c13c0f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c0a1a50 con 0x7f9d6c0a2410 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c0a11b0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c0a1490 con 0x7f9d6c13c0f0 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 10 monclient(hunting): authenticate will time out at 1170.504372s 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d6a7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9d6c0a2410 0x7f9d6c13a9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38170/0 (socket says 172.21.15.78:38170) 2024-09-15T20:31:49.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d6a7fc640 1 -- 172.21.15.78:0/3767524518 learned_addr learned my addr 172.21.15.78:0/3767524518 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7efc4a084640 1 -- 172.21.15.78:0/3408442727 wait complete. 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient: monmap: 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.659 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 librados: starting msgr at 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 librados: starting objecter 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 Processor -- start 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3315781658 0 0) 0x7f9d6c0a1a50 con 0x7f9d6c0a2410 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient(hunting): my global_id is 4317 2024-09-15T20:31:49.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d54003530 con 0x7f9d6c0a2410 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 -- start start 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 librados: setting wanted keys 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 librados: calling monclient init 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient: init 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient(hunting): picked mon.noname-c con 0x7efc440a9500 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient(hunting): picked mon.noname-b con 0x7efc4413c250 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient(hunting): picked mon.noname-a con 0x7efc4413cd40 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc440a0760 con 0x7efc4413cd40 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc4409dd50 con 0x7efc4413c250 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc440a3610 con 0x7efc440a9500 2024-09-15T20:31:49.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc4a084640 10 monclient(hunting): authenticate will time out at 1170.506070s 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2808908779 0 0) 0x7f9d54003530 con 0x7f9d6c0a2410 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 >> v1:172.21.15.78:6790/0 conn(0x7f9d6c13c0f0 legacy=0x7f9d6c13e4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 >> v1:172.21.15.146:6789/0 conn(0x7f9d6c09b3d0 legacy=0x7f9d6c09b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 1 monclient: found mon.noname-a 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c0a41f0 con 0x7f9d6c0a2410 2024-09-15T20:31:49.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d5c0016a0 con 0x7f9d6c0a2410 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d69ffb640 10 monclient: dump: 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.663 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.664 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 5 monclient: authenticate success, global_id 4317 2024-09-15T20:31:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7f9d71488640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc43fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efc4413cd40 0x7efc441e19c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38174/0 (socket says 172.21.15.78:38174) 2024-09-15T20:31:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc43fff640 1 -- 172.21.15.78:0/1254286899 learned_addr learned my addr 172.21.15.78:0/1254286899 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 client.?.objecter ms_handle_connect 0x7efc4413cd40 2024-09-15T20:31:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 client.?.objecter ms_handle_connect 0x7efc4413c250 2024-09-15T20:31:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833128930 0 0) 0x7efc4409dd50 con 0x7efc4413c250 2024-09-15T20:31:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 monclient(hunting): my global_id is 4324 2024-09-15T20:31:49.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc2c003530 con 0x7efc4413c250 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2708725662 0 0) 0x7feac4139cd0 con 0x7feac413e5f0 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 >> v1:172.21.15.78:6790/0 conn(0x7feac40a9610 legacy=0x7feac4138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 >> v1:172.21.15.146:6789/0 conn(0x7feac4137ce0 legacy=0x7feac41395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 monclient: found mon.a 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feac413c640 con 0x7feac413e5f0 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.615+0000 7feaabfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 5 monclient: authenticate success, global_id 4314 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 10 monclient: _renew_subs 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feac413c360 con 0x7feac413e5f0 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feabc001f80 con 0x7feac413e5f0 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feabc003b40 con 0x7feac413e5f0 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 10 monclient: dump: 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.667 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac17fa640 4 set_mon_vals no callback set 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 10 monclient: _renew_subs 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feac413cf70 con 0x7feac413e5f0 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 1 librados: init done 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7feabc003ce0 con 0x7feac413e5f0 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feac40a0a00 con 0x7feac413e5f0 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7feabc048e00 con 0x7feac413e5f0 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 10 client.4314.objecter ms_dispatch 0x7feac4000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 3 client.4314.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 3 client.4314.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 20 client.4314.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feac2ffd640 10 monclient: get_auth_request con 0x7feab0046d50 auth_method 0 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1372312807 0 0) 0x7efc2c003530 con 0x7efc4413c250 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efc4409dd50 con 0x7efc4413c250 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efc34001940 con 0x7efc4413c250 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 monclient(hunting): dump: 2024-09-15T20:31:49.669 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7efc2bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.619+0000 7feaabfff640 10 client.4314.objecter ms_handle_connect 0x7feab0046d50 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1598591859 0 0) 0x7efc4409dd50 con 0x7efc4413c250 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 >> v1:172.21.15.78:6790/0 conn(0x7efc440a9500 legacy=0x7efc4409cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 >> v1:172.21.15.78:6789/0 conn(0x7efc4413cd40 legacy=0x7efc441e19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 monclient: found mon.b 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc440a0480 con 0x7efc4413c250 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 client.?.objecter ms_handle_connect 0x7efc440a9500 2024-09-15T20:31:49.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 5 monclient: authenticate success, global_id 4324 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 10 monclient: _renew_subs 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efc44128680 con 0x7efc4413c250 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 10 client.4324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 10 monclient: _renew_subs 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efc441e30e0 con 0x7efc4413c250 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 1 librados: init done 2024-09-15T20:31:49.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fbf8001af10 con 0x7fbf900a9610 2024-09-15T20:31:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7fbf8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7fbf8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:49.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efc34003620 con 0x7efc4413c250 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc440a14b0 con 0x7efc4413c250 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efc34003a70 con 0x7efc4413c250 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 monclient: dump: 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.674 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc41ffb640 4 set_mon_vals no callback set 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7efc34003c10 con 0x7efc4413c250 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7efc34049e40 con 0x7efc4413c250 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 client.4324.objecter ms_dispatch 0x7efc44000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 3 client.4324.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 3 client.4324.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 20 client.4324.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc437fe640 10 monclient: get_auth_request con 0x7efc2c03e2b0 auth_method 0 2024-09-15T20:31:49.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7efc2bfff640 10 client.4324.objecter ms_handle_connect 0x7efc2c03e2b0 2024-09-15T20:31:49.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7feabc015980 con 0x7feac413e5f0 2024-09-15T20:31:49.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7feaabfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7feaabfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d5c001840 con 0x7f9d6c0a2410 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7f9d69ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7f9d69ffb640 1 -- 172.21.15.78:0/3767524518 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d5c001cd0 con 0x7f9d6c0a2410 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7f9d69ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7f9d69ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7f9d69ffb640 10 monclient: dump: 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7f9d71488640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.623+0000 7f9d71488640 4 set_mon_vals no callback set 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient: shutdown 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- 172.21.15.78:0/3767524518 >> v1:172.21.15.78:6789/0 conn(0x7f9d6c0a2410 legacy=0x7f9d6c13a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- 172.21.15.78:0/3767524518 shutdown_connections 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- 172.21.15.78:0/3767524518 >> 172.21.15.78:0/3767524518 conn(0x7f9d6c0a4c90 msgr2=0x7f9d6c0a50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- 172.21.15.78:0/3767524518 shutdown_connections 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- 172.21.15.78:0/3767524518 wait complete. 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient: build_initial_monmap 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient: monmap: 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 librados: starting msgr at 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 librados: starting objecter 2024-09-15T20:31:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 Processor -- start 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- start start 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 librados: setting wanted keys 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 librados: calling monclient init 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient: init 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient(hunting): picked mon.noname-a con 0x7f9d6c09b3d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient(hunting): picked mon.noname-b con 0x7f9d6c13c0f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient(hunting): picked mon.noname-c con 0x7f9d6c073490 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c0749b0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c09f380 con 0x7f9d6c13c0f0 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d6c1429a0 con 0x7f9d6c073490 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d71488640 10 monclient(hunting): authenticate will time out at 1170.515261s 2024-09-15T20:31:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d6affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9d6c09b3d0 0x7f9d6c141a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:38190/0 (socket says 172.21.15.78:38190) 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d6affd640 1 -- 172.21.15.78:0/25426232 learned_addr learned my addr 172.21.15.78:0/25426232 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 client.?.objecter ms_handle_connect 0x7f9d6c09b3d0 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 client.?.objecter ms_handle_connect 0x7f9d6c073490 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453835976 0 0) 0x7f9d6c0749b0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 monclient(hunting): my global_id is 4320 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d58004ef0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 client.?.objecter ms_handle_connect 0x7f9d6c13c0f0 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.627+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3255937505 0 0) 0x7f9d6c1429a0 con 0x7f9d6c073490 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): my global_id is 4325 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d6c0749b0 con 0x7f9d6c073490 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2506424108 0 0) 0x7f9d58004ef0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d6c1429a0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d60004090 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): dump: 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.683 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 786335447 0 0) 0x7f9d6c0749b0 con 0x7f9d6c073490 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d58004ef0 con 0x7f9d6c073490 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d64001b20 con 0x7f9d6c073490 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient(hunting): dump: 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.684 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 547228291 0 0) 0x7f9d6c1429a0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 >> v1:172.21.15.78:6790/0 conn(0x7f9d6c073490 legacy=0x7f9d6c138a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 >> v1:172.21.15.146:6789/0 conn(0x7f9d6c13c0f0 legacy=0x7f9d6c138370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 monclient: found mon.a 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d6c072930 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d60000ee0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d60001270 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 monclient: dump: 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:49.686 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d697fa640 4 set_mon_vals no callback set 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 5 monclient: authenticate success, global_id 4320 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 10 monclient: _renew_subs 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9d6c142170 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 10 monclient: _renew_subs 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9d6c139190 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 1 librados: init done 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d6c0008e0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f9d60001410 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f9d6004aec0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 10 client.4320.objecter ms_dispatch 0x7f9d6c000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 3 client.4320.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 3 client.4320.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.631+0000 7f9d537fe640 20 client.4320.objecter dump_active .. 0 homeless 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.635+0000 7f9d6a7fc640 10 monclient: get_auth_request con 0x7f9d5803e1b0 auth_method 0 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.635+0000 7f9d537fe640 10 client.4320.objecter ms_handle_connect 0x7f9d5803e1b0 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.635+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7efc34015780 con 0x7efc4413c250 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.635+0000 7efc2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.635+0000 7efc2bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:49.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.639+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f9d600168c0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.639+0000 7f9d537fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:49.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.639+0000 7f9d537fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:49.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.711+0000 7ff9157d8640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T20:31:49.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.711+0000 7ff9157d8640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T20:31:49.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.711+0000 7ff9157d8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.711+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7ff91013fab0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.711+0000 7ff8f77fe640 1 -- 172.21.15.78:0/2382589711 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 956180462) 0x7ff9000161e0 con 0x7ff9100a3ea0 2024-09-15T20:31:49.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.711+0000 7ff8f77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T20:31:49.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.711+0000 7ff8f77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:49.716 INFO:teuthology.orchestra.run.smithi078.stdout:51539607555 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 10 librados: watch_flush enter 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 10 librados: watch_flush exit 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 20 client.4302.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 10 client.4302.objecter successfully canceled tick 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 >> v1:172.21.15.146:6812/435598642 conn(0x7ff8fc03e040 legacy=0x7ff8fc040500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 10 monclient: shutdown 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 >> v1:172.21.15.78:6789/0 conn(0x7ff9100a3ea0 legacy=0x7ff9101d1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 shutdown_connections 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 >> 172.21.15.78:0/2382589711 conn(0x7ff9100a3380 msgr2=0x7ff910130df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 shutdown_connections 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 1 -- 172.21.15.78:0/2382589711 wait complete. 2024-09-15T20:31:49.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.715+0000 7ff9157d8640 1 librados: shutdown 2024-09-15T20:31:49.733 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.5 2024-09-15T20:31:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.759+0000 7fb23021f640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T20:31:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.759+0000 7fb23021f640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T20:31:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.759+0000 7fb23021f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.759+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fb210005380 con 0x7fb228137ce0 2024-09-15T20:31:49.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb21e7fc640 1 -- 172.21.15.78:0/111085953 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 727498649) 0x7fb218015170 con 0x7fb228137ce0 2024-09-15T20:31:49.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb21e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T20:31:49.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb21e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:49.766 INFO:teuthology.orchestra.run.smithi078.stdout:51539607554 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 10 librados: watch_flush enter 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 10 librados: watch_flush exit 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 20 client.4299.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 10 client.4299.objecter successfully canceled tick 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 >> v1:172.21.15.146:6812/435598642 conn(0x7fb20c03e350 legacy=0x7fb20c040810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 10 monclient: shutdown 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 >> v1:172.21.15.78:6789/0 conn(0x7fb228137ce0 legacy=0x7fb2281d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 shutdown_connections 2024-09-15T20:31:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 >> 172.21.15.78:0/111085953 conn(0x7fb2280aa6c0 msgr2=0x7fb228073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 shutdown_connections 2024-09-15T20:31:49.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 1 -- 172.21.15.78:0/111085953 wait complete. 2024-09-15T20:31:49.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.763+0000 7fb23021f640 1 librados: shutdown 2024-09-15T20:31:49.789 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607554 for osd.0 2024-09-15T20:31:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T20:31:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T20:31:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fbf74005380 con 0x7fbf900a9610 2024-09-15T20:31:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf8cff9640 1 -- 172.21.15.78:0/238690155 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 1444633209) 0x7fbf8001fe50 con 0x7fbf900a9610 2024-09-15T20:31:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T20:31:49.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf8cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:49.803 INFO:teuthology.orchestra.run.smithi078.stdout:47244640260 2024-09-15T20:31:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 10 librados: watch_flush enter 2024-09-15T20:31:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 10 librados: watch_flush exit 2024-09-15T20:31:49.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 20 client.4319.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 10 client.4319.objecter successfully canceled tick 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 >> v1:172.21.15.146:6812/435598642 conn(0x7fbf7003e0d0 legacy=0x7fbf70040590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 10 monclient: shutdown 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 >> v1:172.21.15.78:6790/0 conn(0x7fbf900a9610 legacy=0x7fbf90138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 shutdown_connections 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 >> 172.21.15.78:0/238690155 conn(0x7fbf900aa6c0 msgr2=0x7fbf90074920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.799+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 shutdown_connections 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.803+0000 7fbf961c7640 1 -- 172.21.15.78:0/238690155 wait complete. 2024-09-15T20:31:49.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.803+0000 7fbf961c7640 1 librados: shutdown 2024-09-15T20:31:49.820 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.3 2024-09-15T20:31:49.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.875+0000 7f9d71488640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T20:31:49.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.875+0000 7f9d71488640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T20:31:49.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.875+0000 7f9d71488640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.875+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f9d6c0745c0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.875+0000 7f9d537fe640 1 -- 172.21.15.78:0/25426232 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 956180462) 0x7f9d600161c0 con 0x7f9d6c09b3d0 2024-09-15T20:31:49.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.875+0000 7f9d537fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T20:31:49.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.875+0000 7f9d537fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:49.882 INFO:teuthology.orchestra.run.smithi078.stdout:51539607555 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 10 librados: watch_flush enter 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 10 librados: watch_flush exit 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 20 client.4320.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 10 client.4320.objecter successfully canceled tick 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 >> v1:172.21.15.146:6812/435598642 conn(0x7f9d5803e1b0 legacy=0x7f9d58040670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 10 monclient: shutdown 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 >> v1:172.21.15.78:6789/0 conn(0x7f9d6c09b3d0 legacy=0x7f9d6c141a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 shutdown_connections 2024-09-15T20:31:49.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 >> 172.21.15.78:0/25426232 conn(0x7f9d6c0a4c90 msgr2=0x7f9d6c09f590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 shutdown_connections 2024-09-15T20:31:49.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 1 -- 172.21.15.78:0/25426232 wait complete. 2024-09-15T20:31:49.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.879+0000 7f9d71488640 1 librados: shutdown 2024-09-15T20:31:49.897 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607555 for osd.2 2024-09-15T20:31:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.923+0000 7feac997c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T20:31:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.923+0000 7feac997c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T20:31:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.923+0000 7feac997c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.923+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7feac406a080 con 0x7feac413e5f0 2024-09-15T20:31:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.923+0000 7feaabfff640 1 -- 172.21.15.78:0/2964418763 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 727498649) 0x7feabc019d60 con 0x7feac413e5f0 2024-09-15T20:31:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.923+0000 7feaabfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T20:31:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.923+0000 7feaabfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:49.930 INFO:teuthology.orchestra.run.smithi078.stdout:51539607554 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 10 librados: watch_flush enter 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 10 librados: watch_flush exit 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 20 client.4314.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 10 client.4314.objecter successfully canceled tick 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 >> v1:172.21.15.146:6812/435598642 conn(0x7feab0046d50 legacy=0x7feab0049210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 10 monclient: shutdown 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 >> v1:172.21.15.78:6789/0 conn(0x7feac413e5f0 legacy=0x7feac41d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 shutdown_connections 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 >> 172.21.15.78:0/2964418763 conn(0x7feac40aa6c0 msgr2=0x7feac4073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 shutdown_connections 2024-09-15T20:31:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 1 -- 172.21.15.78:0/2964418763 wait complete. 2024-09-15T20:31:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.927+0000 7feac997c640 1 librados: shutdown 2024-09-15T20:31:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.935+0000 7efc4a084640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T20:31:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.935+0000 7efc4a084640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T20:31:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.935+0000 7efc4a084640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:31:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.935+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7efc4406a080 con 0x7efc4413c250 2024-09-15T20:31:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.935+0000 7efc2bfff640 1 -- 172.21.15.78:0/1254286899 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1444633209) 0x7efc34019b60 con 0x7efc4413c250 2024-09-15T20:31:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.935+0000 7efc2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T20:31:49.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.935+0000 7efc2bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:49.940 INFO:teuthology.orchestra.run.smithi078.stdout:47244640260 2024-09-15T20:31:49.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 10 librados: watch_flush enter 2024-09-15T20:31:49.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 10 librados: watch_flush exit 2024-09-15T20:31:49.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 20 client.4324.objecter shutdown clearing up homeless session... 2024-09-15T20:31:49.942 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 10 client.4324.objecter successfully canceled tick 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 >> v1:172.21.15.146:6812/435598642 conn(0x7efc2c03e2b0 legacy=0x7efc2c040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 10 monclient: shutdown 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 >> v1:172.21.15.146:6789/0 conn(0x7efc4413c250 legacy=0x7efc4409d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 shutdown_connections 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 >> 172.21.15.78:0/1254286899 conn(0x7efc440aa6c0 msgr2=0x7efc440a2590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 shutdown_connections 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 1 -- 172.21.15.78:0/1254286899 wait complete. 2024-09-15T20:31:49.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:49.939+0000 7efc4a084640 1 librados: shutdown 2024-09-15T20:31:49.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607554 for osd.4 2024-09-15T20:31:49.957 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640260 for osd.1 2024-09-15T20:31:50.734 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-09-15T20:31:50.790 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-09-15T20:31:50.821 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.855+0000 7fcc3891a640 10 monclient: get_monmap_and_config 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.855+0000 7fcc3891a640 10 monclient: build_initial_monmap 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.855+0000 7fcc3891a640 10 monclient: monmap: 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:50.860533+0000 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:50.860533+0000 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.861 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:50.862 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:50.862 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:50.862 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 Processor -- start 2024-09-15T20:31:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- start start 2024-09-15T20:31:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient: init 2024-09-15T20:31:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:50.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient(hunting): picked mon.noname-c con 0x7fcc34137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient(hunting): picked mon.noname-b con 0x7fcc340a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient(hunting): picked mon.noname-a con 0x7fcc3413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a6420 con 0x7fcc3413e5f0 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a6700 con 0x7fcc34137ce0 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc340a2110 con 0x7fcc340a9610 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient(hunting): authenticate will time out at 1171.744802s 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc31d9b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcc340a9610 0x7fcc3413ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54616/0 (socket says 172.21.15.78:54616) 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc31d9b640 1 -- 172.21.15.78:0/3386988815 learned_addr learned my addr 172.21.15.78:0/3386988815 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 129761718 0 0) 0x7fcc340a2110 con 0x7fcc340a9610 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient(hunting): my global_id is 4328 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc1c003530 con 0x7fcc340a9610 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 368220247 0 0) 0x7fcc340a6420 con 0x7fcc3413e5f0 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient(hunting): my global_id is 4323 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc340a2110 con 0x7fcc3413e5f0 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1966530946 0 0) 0x7fcc1c003530 con 0x7fcc340a9610 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 >> v1:172.21.15.146:6789/0 conn(0x7fcc34137ce0 legacy=0x7fcc341380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 >> v1:172.21.15.78:6789/0 conn(0x7fcc3413e5f0 legacy=0x7fcc341409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: _finish_hunting 0 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 monclient: found mon.noname-b 2024-09-15T20:31:50.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340a4bc0 con 0x7fcc340a9610 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: _finish_auth 0 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: _check_auth_tickets 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcc280016a0 con 0x7fcc340a9610 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 5 monclient: authenticate success, global_id 4328 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: dump: 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.867 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc28001840 con 0x7fcc340a9610 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 1 -- 172.21.15.78:0/3386988815 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcc28001c50 con 0x7fcc340a9610 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:50.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3159a640 10 monclient: dump: 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 4 set_mon_vals no callback set 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient: shutdown 2024-09-15T20:31:50.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- 172.21.15.78:0/3386988815 >> v1:172.21.15.78:6790/0 conn(0x7fcc340a9610 legacy=0x7fcc3413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- 172.21.15.78:0/3386988815 shutdown_connections 2024-09-15T20:31:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- 172.21.15.78:0/3386988815 >> 172.21.15.78:0/3386988815 conn(0x7fcc340aa6c0 msgr2=0x7fcc340aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- 172.21.15.78:0/3386988815 shutdown_connections 2024-09-15T20:31:50.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 1 -- 172.21.15.78:0/3386988815 wait complete. 2024-09-15T20:31:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient: build_initial_monmap 2024-09-15T20:31:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.859+0000 7fcc3891a640 10 monclient: monmap: 2024-09-15T20:31:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:50.874 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 librados: starting msgr at 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 librados: starting objecter 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 Processor -- start 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 -- start start 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 librados: setting wanted keys 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 librados: calling monclient init 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: init 2024-09-15T20:31:50.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient(hunting): picked mon.noname-b con 0x7fcc340a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient(hunting): picked mon.noname-c con 0x7fcc34137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient(hunting): picked mon.noname-a con 0x7fcc3413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc34074330 con 0x7fcc3413e5f0 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc3413c0e0 con 0x7fcc340a9610 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcc3413ab00 con 0x7fcc34137ce0 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient(hunting): authenticate will time out at 1171.749585s 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc31d9b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcc34137ce0 0x7fcc341d1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54630/0 (socket says 172.21.15.78:54630) 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc32d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcc3413e5f0 0x7fcc341d5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50784/0 (socket says 172.21.15.78:50784) 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc31d9b640 1 -- 172.21.15.78:0/3783404322 learned_addr learned my addr 172.21.15.78:0/3783404322 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 client.?.objecter ms_handle_connect 0x7fcc34137ce0 2024-09-15T20:31:50.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 client.?.objecter ms_handle_connect 0x7fcc3413e5f0 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 client.?.objecter ms_handle_connect 0x7fcc340a9610 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3605255818 0 0) 0x7fcc2c001d00 con 0x7fcc3413e5f0 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): my global_id is 4326 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc14004ec0 con 0x7fcc3413e5f0 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 133107833 0 0) 0x7fcc3413ab00 con 0x7fcc34137ce0 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): my global_id is 4331 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc2c001d00 con 0x7fcc34137ce0 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2617249055 0 0) 0x7fcc3413c0e0 con 0x7fcc340a9610 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): my global_id is 4330 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcc3413ab00 con 0x7fcc340a9610 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1872888496 0 0) 0x7fcc2c001d00 con 0x7fcc34137ce0 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc3413c0e0 con 0x7fcc34137ce0 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcc28001c10 con 0x7fcc34137ce0 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:50.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): dump: 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 619433110 0 0) 0x7fcc14004ec0 con 0x7fcc3413e5f0 2024-09-15T20:31:50.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcc2c001d00 con 0x7fcc3413e5f0 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcc2c001ac0 con 0x7fcc3413e5f0 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient(hunting): dump: 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.880 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2687249375 0 0) 0x7fcc3413c0e0 con 0x7fcc34137ce0 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 >> v1:172.21.15.146:6789/0 conn(0x7fcc340a9610 legacy=0x7fcc341c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 >> v1:172.21.15.78:6789/0 conn(0x7fcc3413e5f0 legacy=0x7fcc341d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 monclient: found mon.c 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc340a91e0 con 0x7fcc34137ce0 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient: _finish_auth 0 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 5 monclient: authenticate success, global_id 4331 2024-09-15T20:31:50.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: _renew_subs 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcc3413be00 con 0x7fcc34137ce0 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcc28004aa0 con 0x7fcc34137ce0 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 client.4331.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcc28004df0 con 0x7fcc34137ce0 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc22ffd640 10 monclient: dump: 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.882 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: _renew_subs 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcc340a0230 con 0x7fcc34137ce0 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc30d99640 4 set_mon_vals no callback set 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 librados: init done 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.863+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc3406a080 con 0x7fcc34137ce0 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.867+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fcc28010cc0 con 0x7fcc34137ce0 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.867+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7fcc2804ad30 con 0x7fcc34137ce0 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.867+0000 7fcc22ffd640 10 client.4331.objecter ms_dispatch 0x7fcc34000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.867+0000 7fcc22ffd640 3 client.4331.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.867+0000 7fcc22ffd640 3 client.4331.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.867+0000 7fcc22ffd640 20 client.4331.objecter dump_active .. 0 homeless 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.867+0000 7fcc3259c640 10 monclient: get_auth_request con 0x7fcc1403e010 auth_method 0 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.867+0000 7fcc22ffd640 10 client.4331.objecter ms_handle_connect 0x7fcc1403e010 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.875+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fcc28016790 con 0x7fcc34137ce0 2024-09-15T20:31:50.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.875+0000 7fcc22ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.875+0000 7fcc22ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:50.898 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-09-15T20:31:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 10 monclient: get_monmap_and_config 2024-09-15T20:31:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 10 monclient: build_initial_monmap 2024-09-15T20:31:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 10 monclient: monmap: 2024-09-15T20:31:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:50.911775+0000 2024-09-15T20:31:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:50.911775+0000 2024-09-15T20:31:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:50.913 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 1 Processor -- start 2024-09-15T20:31:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 1 -- start start 2024-09-15T20:31:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 10 monclient: init 2024-09-15T20:31:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 10 monclient(hunting): picked mon.noname-b con 0x7ffba4137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.907+0000 7ffbaaf77640 10 monclient(hunting): picked mon.noname-a con 0x7ffba40a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient(hunting): picked mon.noname-c con 0x7ffba413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffba40a6420 con 0x7ffba40a9610 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffba40a6700 con 0x7ffba413e5f0 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffba40a2110 con 0x7ffba4137ce0 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient(hunting): authenticate will time out at 1171.796067s 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ffba40a9610 0x7ffba413ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50786/0 (socket says 172.21.15.78:50786) 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba3fff640 1 -- 172.21.15.78:0/1946652277 learned_addr learned my addr 172.21.15.78:0/1946652277 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2103038235 0 0) 0x7ffba40a6420 con 0x7ffba40a9610 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient(hunting): my global_id is 4329 2024-09-15T20:31:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb94003530 con 0x7ffba40a9610 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2635648089 0 0) 0x7ffba40a2110 con 0x7ffba4137ce0 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient(hunting): my global_id is 4334 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffba40a6420 con 0x7ffba4137ce0 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1029239172 0 0) 0x7ffb94003530 con 0x7ffba40a9610 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 >> v1:172.21.15.78:6790/0 conn(0x7ffba4137ce0 legacy=0x7ffba41380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 >> v1:172.21.15.146:6789/0 conn(0x7ffba413e5f0 legacy=0x7ffba41409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 monclient: found mon.noname-a 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba40a4bc0 con 0x7ffba40a9610 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ffb900016a0 con 0x7ffba40a9610 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: dump: 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 5 monclient: authenticate success, global_id 4329 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb900018c0 con 0x7ffba40a9610 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 1 -- 172.21.15.78:0/1946652277 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ffb90001c90 con 0x7ffba40a9610 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 4 set_mon_vals no callback set 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba37fe640 10 monclient: dump: 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient: shutdown 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- 172.21.15.78:0/1946652277 >> v1:172.21.15.78:6789/0 conn(0x7ffba40a9610 legacy=0x7ffba413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- 172.21.15.78:0/1946652277 shutdown_connections 2024-09-15T20:31:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- 172.21.15.78:0/1946652277 >> 172.21.15.78:0/1946652277 conn(0x7ffba40aa6c0 msgr2=0x7ffba40aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- 172.21.15.78:0/1946652277 shutdown_connections 2024-09-15T20:31:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- 172.21.15.78:0/1946652277 wait complete. 2024-09-15T20:31:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient: build_initial_monmap 2024-09-15T20:31:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient: monmap: 2024-09-15T20:31:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 librados: starting msgr at 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 librados: starting objecter 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 Processor -- start 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- start start 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 librados: setting wanted keys 2024-09-15T20:31:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 librados: calling monclient init 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient: init 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient(hunting): picked mon.noname-b con 0x7ffba40a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient(hunting): picked mon.noname-a con 0x7ffba4137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient(hunting): picked mon.noname-c con 0x7ffba413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffba40a14b0 con 0x7ffba4137ce0 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffba409f970 con 0x7ffba40a9610 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffba40a0160 con 0x7ffba413e5f0 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffbaaf77640 10 monclient(hunting): authenticate will time out at 1171.799193s 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba9514640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ffba413e5f0 0x7ffba41d5870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54642/0 (socket says 172.21.15.78:54642) 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba9514640 1 -- 172.21.15.78:0/3766394966 learned_addr learned my addr 172.21.15.78:0/3766394966 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 client.?.objecter ms_handle_connect 0x7ffba413e5f0 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 client.?.objecter ms_handle_connect 0x7ffba4137ce0 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2990152626 0 0) 0x7ffba40a14b0 con 0x7ffba4137ce0 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): my global_id is 4332 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffb88004ec0 con 0x7ffba4137ce0 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 210824867 0 0) 0x7ffba40a0160 con 0x7ffba413e5f0 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): my global_id is 4337 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffba40a14b0 con 0x7ffba413e5f0 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 client.?.objecter ms_handle_connect 0x7ffba40a9610 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2508974566 0 0) 0x7ffb88004ec0 con 0x7ffba4137ce0 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffba40a0160 con 0x7ffba4137ce0 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ffb90001c10 con 0x7ffba4137ce0 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): dump: 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.928 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2944931755 0 0) 0x7ffba40a14b0 con 0x7ffba413e5f0 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ffb88004ec0 con 0x7ffba413e5f0 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2499487791 0 0) 0x7ffba409f970 con 0x7ffba40a9610 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): my global_id is 4336 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffba40a14b0 con 0x7ffba40a9610 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ffb9c001c80 con 0x7ffba413e5f0 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.911+0000 7ffba17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient(hunting): dump: 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.930 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3280223844 0 0) 0x7ffb88004ec0 con 0x7ffba413e5f0 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 >> v1:172.21.15.146:6789/0 conn(0x7ffba40a9610 legacy=0x7ffba41cd160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 >> v1:172.21.15.78:6789/0 conn(0x7ffba4137ce0 legacy=0x7ffba41d2140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 monclient: found mon.c 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba4137770 con 0x7ffba413e5f0 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:50.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 5 monclient: authenticate success, global_id 4337 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 10 monclient: _renew_subs 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ffba409f200 con 0x7ffba413e5f0 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 10 client.4337.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 10 monclient: _renew_subs 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ffba409dea0 con 0x7ffba413e5f0 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 1 librados: init done 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ffb9c0039a0 con 0x7ffba413e5f0 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ffb9c003df0 con 0x7ffba413e5f0 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 monclient: dump: 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ffb9c004050 con 0x7ffba413e5f0 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba2ffd640 4 set_mon_vals no callback set 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7ffb9c04b360 con 0x7ffba413e5f0 2024-09-15T20:31:50.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 client.4337.objecter ms_dispatch 0x7ffba4000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 3 client.4337.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 3 client.4337.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 20 client.4337.objecter dump_active .. 0 homeless 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb84005090 con 0x7ffba413e5f0 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba8d13640 10 monclient: get_auth_request con 0x7ffb8803e300 auth_method 0 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.915+0000 7ffba17fa640 10 client.4337.objecter ms_handle_connect 0x7ffb8803e300 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.919+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7ffb9c016d60 con 0x7ffba413e5f0 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.919+0000 7ffba17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.919+0000 7ffba17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:50.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-09-15T20:31:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient: get_monmap_and_config 2024-09-15T20:31:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient: build_initial_monmap 2024-09-15T20:31:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient: monmap: 2024-09-15T20:31:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:50.953662+0000 2024-09-15T20:31:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:50.953662+0000 2024-09-15T20:31:50.954 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:50.955 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.955 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:50.955 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:50.955 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:50.955 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 1 Processor -- start 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 1 -- start start 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient: init 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient(hunting): picked mon.noname-b con 0x7f1eac0a1a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient(hunting): picked mon.noname-a con 0x7f1eac0a2f80 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient(hunting): picked mon.noname-c con 0x7f1eac0a2500 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac09e5b0 con 0x7f1eac0a2f80 2024-09-15T20:31:50.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac09dc90 con 0x7f1eac0a2500 2024-09-15T20:31:50.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac09ef90 con 0x7f1eac0a1a10 2024-09-15T20:31:50.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:50.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient(hunting): authenticate will time out at 1171.838677s 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eaaffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1eac0a1a10 0x7f1eac0a1df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54650/0 (socket says 172.21.15.78:54650) 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eaa7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1eac0a2f80 0x7f1eac139d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50800/0 (socket says 172.21.15.78:50800) 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eaaffd640 1 -- 172.21.15.78:0/1226339419 learned_addr learned my addr 172.21.15.78:0/1226339419 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3862334352 0 0) 0x7f1eac09ef90 con 0x7f1eac0a1a10 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient(hunting): my global_id is 4340 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e94005740 con 0x7f1eac0a1a10 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4055452650 0 0) 0x7f1eac09e5b0 con 0x7f1eac0a2f80 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient(hunting): my global_id is 4335 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eac09ef90 con 0x7f1eac0a2f80 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3533802189 0 0) 0x7f1e94005740 con 0x7f1eac0a1a10 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 >> v1:172.21.15.146:6789/0 conn(0x7f1eac0a2500 legacy=0x7f1eac13d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 >> v1:172.21.15.78:6789/0 conn(0x7f1eac0a2f80 legacy=0x7f1eac139d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 monclient: found mon.noname-b 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eac141bf0 con 0x7f1eac0a1a10 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:50.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1e9c0016a0 con 0x7f1eac0a1a10 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: dump: 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.960 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 5 monclient: authenticate success, global_id 4340 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1e9c001840 con 0x7f1eac0a1a10 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 1 -- 172.21.15.78:0/1226339419 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1e9c001c50 con 0x7f1eac0a1a10 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1ea9ffb640 10 monclient: dump: 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.961 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.951+0000 7f1eb1939640 4 set_mon_vals no callback set 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient: shutdown 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:50.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- 172.21.15.78:0/1226339419 >> v1:172.21.15.78:6790/0 conn(0x7f1eac0a1a10 legacy=0x7f1eac0a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.962 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-09-15T20:31:50.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- 172.21.15.78:0/1226339419 shutdown_connections 2024-09-15T20:31:50.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- 172.21.15.78:0/1226339419 >> 172.21.15.78:0/1226339419 conn(0x7f1eac0a3ce0 msgr2=0x7f1eac0a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- 172.21.15.78:0/1226339419 shutdown_connections 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- 172.21.15.78:0/1226339419 wait complete. 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient: build_initial_monmap 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient: monmap: 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:50.969 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 librados: starting msgr at 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 librados: starting objecter 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 Processor -- start 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- start start 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 librados: setting wanted keys 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 librados: calling monclient init 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient: init 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient(hunting): picked mon.noname-c con 0x7f1eac0a1a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eaaffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1eac0a1a10 0x7f1eac1386a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54656/0 (socket says 172.21.15.78:54656) 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eaaffd640 1 -- 172.21.15.78:0/2128361738 learned_addr learned my addr 172.21.15.78:0/2128361738 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:50.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient(hunting): picked mon.noname-a con 0x7f1eac0a2500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 client.?.objecter ms_handle_connect 0x7f1eac0a1a10 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 client.?.objecter ms_handle_connect 0x7f1eac0a2500 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient(hunting): picked mon.noname-b con 0x7f1eac0a2f80 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac09e200 con 0x7f1eac0a2500 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac13b7b0 con 0x7f1eac0a2f80 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1eac1409d0 con 0x7f1eac0a1a10 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1eb1939640 10 monclient(hunting): authenticate will time out at 1171.843779s 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2042535974 0 0) 0x7f1eac09e200 con 0x7f1eac0a2500 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 monclient(hunting): my global_id is 4338 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e98004ec0 con 0x7f1eac0a2500 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2921891126 0 0) 0x7f1eac1409d0 con 0x7f1eac0a1a10 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.955+0000 7f1e93fff640 10 monclient(hunting): my global_id is 4343 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1eac09e200 con 0x7f1eac0a1a10 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 client.?.objecter ms_handle_connect 0x7f1eac0a2f80 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 409583549 0 0) 0x7f1e98004ec0 con 0x7f1eac0a2500 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1eac1409d0 con 0x7f1eac0a2500 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1ea0001940 con 0x7f1eac0a2500 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:50.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient(hunting): dump: 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2232432745 0 0) 0x7f1eac13b7b0 con 0x7f1eac0a2f80 2024-09-15T20:31:50.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient(hunting): my global_id is 4342 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1e98004ec0 con 0x7f1eac0a2f80 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2974267360 0 0) 0x7f1eac1409d0 con 0x7f1eac0a2500 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 >> v1:172.21.15.78:6790/0 conn(0x7f1eac0a1a10 legacy=0x7f1eac1386a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 >> v1:172.21.15.146:6789/0 conn(0x7f1eac0a2f80 legacy=0x7f1eac1394c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 monclient: found mon.a 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eac13a160 con 0x7f1eac0a2500 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient: _finish_auth 0 2024-09-15T20:31:50.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 5 monclient: authenticate success, global_id 4338 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 10 monclient: _renew_subs 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1eac13bc20 con 0x7f1eac0a2500 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ea00047f0 con 0x7f1eac0a2500 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 10 monclient: _renew_subs 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1eac1d1b50 con 0x7f1eac0a2500 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eb1939640 1 librados: init done 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1ea0004c40 con 0x7f1eac0a2500 2024-09-15T20:31:50.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1ea97fa640 4 set_mon_vals no callback set 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 monclient: dump: 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e91ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e91ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e91ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e91ffb640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1e84005090 con 0x7f1eac0a2500 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1ea0004de0 con 0x7f1eac0a2500 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f1ea004af70 con 0x7f1eac0a2500 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 client.4338.objecter ms_dispatch 0x7f1eac000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 3 client.4338.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 3 client.4338.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 20 client.4338.objecter dump_active .. 0 homeless 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1eaaffd640 10 monclient: get_auth_request con 0x7f1e9803df30 auth_method 0 2024-09-15T20:31:50.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.959+0000 7f1e93fff640 10 client.4338.objecter ms_handle_connect 0x7f1e9803df30 2024-09-15T20:31:50.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.967+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1ea0016910 con 0x7f1eac0a2500 2024-09-15T20:31:50.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.967+0000 7f1e93fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:50.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:50.967+0000 7f1e93fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:51.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.023+0000 7f13e6400640 10 monclient: get_monmap_and_config 2024-09-15T20:31:51.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.023+0000 7f13e6400640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.023+0000 7f13e6400640 10 monclient: monmap: 2024-09-15T20:31:51.029 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.029 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.029 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:51.028612+0000 2024-09-15T20:31:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:51.028612+0000 2024-09-15T20:31:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:51.030 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 1 Processor -- start 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 1 -- start start 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient: init 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient(hunting): picked mon.noname-a con 0x7f13e0137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient(hunting): picked mon.noname-b con 0x7f13e00a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient(hunting): picked mon.noname-c con 0x7f13e013e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e00a6420 con 0x7f13e0137ce0 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e00a6700 con 0x7f13e013e5f0 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e00a2110 con 0x7f13e00a9610 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient(hunting): authenticate will time out at 1171.913545s 2024-09-15T20:31:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13dffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f13e0137ce0 0x7f13e01380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50814/0 (socket says 172.21.15.78:50814) 2024-09-15T20:31:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13d7fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f13e00a9610 0x7f13e013ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54668/0 (socket says 172.21.15.78:54668) 2024-09-15T20:31:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13dffff640 1 -- 172.21.15.78:0/1914664017 learned_addr learned my addr 172.21.15.78:0/1914664017 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336606872 0 0) 0x7f13e00a2110 con 0x7f13e00a9610 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient(hunting): my global_id is 4346 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13cc003530 con 0x7f13e00a9610 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1358735104 0 0) 0x7f13e00a6420 con 0x7f13e0137ce0 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient(hunting): my global_id is 4341 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13e00a2110 con 0x7f13e0137ce0 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2519981188 0 0) 0x7f13cc003530 con 0x7f13e00a9610 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 >> v1:172.21.15.146:6789/0 conn(0x7f13e013e5f0 legacy=0x7f13e01409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 >> v1:172.21.15.78:6789/0 conn(0x7f13e0137ce0 legacy=0x7f13e01380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 monclient: found mon.noname-b 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e00a4bc0 con 0x7f13e00a9610 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f13c80016a0 con 0x7f13e00a9610 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 5 monclient: authenticate success, global_id 4346 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: dump: 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.034 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13c8001840 con 0x7f13e00a9610 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 1 -- 172.21.15.78:0/1914664017 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f13c8001c50 con 0x7f13e00a9610 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13df7fe640 10 monclient: dump: 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:51.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 4 set_mon_vals no callback set 2024-09-15T20:31:51.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 10 monclient: shutdown 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.027+0000 7f13e6400640 1 -- 172.21.15.78:0/1914664017 >> v1:172.21.15.78:6790/0 conn(0x7f13e00a9610 legacy=0x7f13e013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 -- 172.21.15.78:0/1914664017 shutdown_connections 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 -- 172.21.15.78:0/1914664017 >> 172.21.15.78:0/1914664017 conn(0x7f13e00aa6c0 msgr2=0x7f13e00aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 -- 172.21.15.78:0/1914664017 shutdown_connections 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 -- 172.21.15.78:0/1914664017 wait complete. 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient: monmap: 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:51.040 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 librados: starting msgr at 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 librados: starting objecter 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 Processor -- start 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 -- start start 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 librados: setting wanted keys 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 librados: calling monclient init 2024-09-15T20:31:51.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient: init 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient(hunting): picked mon.noname-b con 0x7f13e00a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient(hunting): picked mon.noname-c con 0x7f13e0137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient(hunting): picked mon.noname-a con 0x7f13e013e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e00a9270 con 0x7f13e013e5f0 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e0071e60 con 0x7f13e00a9610 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f13e0072140 con 0x7f13e0137ce0 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e6400640 10 monclient(hunting): authenticate will time out at 1171.918321s 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e499d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f13e013e5f0 0x7f13e01d9230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50816/0 (socket says 172.21.15.78:50816) 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13e499d640 1 -- 172.21.15.78:0/2038629713 learned_addr learned my addr 172.21.15.78:0/2038629713 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 client.?.objecter ms_handle_connect 0x7f13e013e5f0 2024-09-15T20:31:51.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1801956178 0 0) 0x7f13d8001650 con 0x7f13e013e5f0 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): my global_id is 4344 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c0004ec0 con 0x7f13e013e5f0 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 client.?.objecter ms_handle_connect 0x7f13e00a9610 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2618343291 0 0) 0x7f13c0004ec0 con 0x7f13e013e5f0 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f13d8001650 con 0x7f13e013e5f0 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f13d8001f20 con 0x7f13e013e5f0 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:51.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): dump: 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1923624467 0 0) 0x7f13e0071e60 con 0x7f13e00a9610 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient(hunting): my global_id is 4348 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f13c0004ec0 con 0x7f13e00a9610 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1566347509 0 0) 0x7f13d8001650 con 0x7f13e013e5f0 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 >> v1:172.21.15.78:6790/0 conn(0x7f13e0137ce0 legacy=0x7f13e006f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 >> v1:172.21.15.146:6789/0 conn(0x7f13e00a9610 legacy=0x7f13e006ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 monclient: found mon.a 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e006fc30 con 0x7f13e013e5f0 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 client.?.objecter ms_handle_connect 0x7f13e0137ce0 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.031+0000 7f13dd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f13d8003c50 con 0x7f13e013e5f0 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13deffd640 4 set_mon_vals no callback set 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f13d8004060 con 0x7f13e013e5f0 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 5 monclient: authenticate success, global_id 4344 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 10 monclient: _renew_subs 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f13e00743e0 con 0x7f13e013e5f0 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 10 client.4344.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 10 monclient: _renew_subs 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f13e0071b80 con 0x7f13e013e5f0 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 1 librados: init done 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 10 monclient: dump: 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.047 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13e006a080 con 0x7f13e013e5f0 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f13d8004200 con 0x7f13e013e5f0 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f13d804b5e0 con 0x7f13e013e5f0 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 10 client.4344.objecter ms_dispatch 0x7f13e0000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 3 client.4344.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 3 client.4344.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:51.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 20 client.4344.objecter dump_active .. 0 homeless 2024-09-15T20:31:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dffff640 10 monclient: get_auth_request con 0x7f13c003e0b0 auth_method 0 2024-09-15T20:31:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.035+0000 7f13dd7fa640 10 client.4344.objecter ms_handle_connect 0x7f13c003e0b0 2024-09-15T20:31:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.043+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f13e00a9270 con 0x7f13e013e5f0 2024-09-15T20:31:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.043+0000 7f13dd7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.043+0000 7f13dd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.059+0000 7fcc3891a640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T20:31:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.059+0000 7fcc3891a640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T20:31:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.059+0000 7fcc3891a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.059+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fcc340a3d30 con 0x7fcc34137ce0 2024-09-15T20:31:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.059+0000 7fcc22ffd640 1 -- 172.21.15.78:0/3783404322 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (unknown 2131975755 0 532354816) 0x7fcc28016090 con 0x7fcc34137ce0 2024-09-15T20:31:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.059+0000 7fcc22ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 5}] 2024-09-15T20:31:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.059+0000 7fcc22ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:51.066 INFO:teuthology.orchestra.run.smithi078.stdout:51539607557 2024-09-15T20:31:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 10 librados: watch_flush enter 2024-09-15T20:31:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 10 librados: watch_flush exit 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 20 client.4331.objecter shutdown clearing up homeless session... 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 10 client.4331.objecter successfully canceled tick 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 >> v1:172.21.15.146:6812/435598642 conn(0x7fcc1403e010 legacy=0x7fcc140404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 10 monclient: shutdown 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 >> v1:172.21.15.78:6790/0 conn(0x7fcc34137ce0 legacy=0x7fcc341d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 shutdown_connections 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 >> 172.21.15.78:0/3783404322 conn(0x7fcc340aa6c0 msgr2=0x7fcc34073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 shutdown_connections 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 1 -- 172.21.15.78:0/3783404322 wait complete. 2024-09-15T20:31:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.063+0000 7fcc3891a640 1 librados: shutdown 2024-09-15T20:31:51.084 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.5 2024-09-15T20:31:51.085 DEBUG:teuthology.parallel:result is None 2024-09-15T20:31:51.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient: get_monmap_and_config 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient: monmap: 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:51.099187+0000 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:51.099187+0000 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:51.100 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:51.101 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 1 Processor -- start 2024-09-15T20:31:51.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 1 -- start start 2024-09-15T20:31:51.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient: init 2024-09-15T20:31:51.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:51.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient(hunting): picked mon.noname-c con 0x7ff134137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient(hunting): picked mon.noname-b con 0x7ff1340a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient(hunting): picked mon.noname-a con 0x7ff13413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff139727640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff1340a9610 0x7ff13413ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54684/0 (socket says 172.21.15.78:54684) 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff139727640 1 -- 172.21.15.78:0/3136492195 learned_addr learned my addr 172.21.15.78:0/3136492195 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 1 -- 172.21.15.78:0/3136492195 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1340a6420 con 0x7ff13413e5f0 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 1 -- 172.21.15.78:0/3136492195 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1340a6700 con 0x7ff134137ce0 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 1 -- 172.21.15.78:0/3136492195 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1340a2110 con 0x7ff1340a9610 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.095+0000 7ff13c18c640 10 monclient(hunting): authenticate will time out at 1171.983726s 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 523318789 0 0) 0x7ff1340a2110 con 0x7ff1340a9610 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient(hunting): my global_id is 4352 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff120003530 con 0x7ff1340a9610 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3366100263 0 0) 0x7ff1340a6420 con 0x7ff13413e5f0 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient(hunting): my global_id is 4347 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1340a2110 con 0x7ff13413e5f0 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4216906351 0 0) 0x7ff1340a2110 con 0x7ff13413e5f0 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 >> v1:172.21.15.78:6790/0 conn(0x7ff1340a9610 legacy=0x7ff13413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 >> v1:172.21.15.146:6789/0 conn(0x7ff134137ce0 legacy=0x7ff1341380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 monclient: found mon.noname-a 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1340a4bc0 con 0x7ff13413e5f0 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff12c001930 con 0x7ff13413e5f0 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: dump: 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.105 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 5 monclient: authenticate success, global_id 4347 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff12c001ad0 con 0x7ff13413e5f0 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 1 -- 172.21.15.78:0/3136492195 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff12c001ee0 con 0x7ff13413e5f0 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff138f26640 10 monclient: dump: 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:51.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 4 set_mon_vals no callback set 2024-09-15T20:31:51.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 10 monclient: shutdown 2024-09-15T20:31:51.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 -- 172.21.15.78:0/3136492195 >> v1:172.21.15.78:6789/0 conn(0x7ff13413e5f0 legacy=0x7ff1341409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 -- 172.21.15.78:0/3136492195 shutdown_connections 2024-09-15T20:31:51.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 -- 172.21.15.78:0/3136492195 >> 172.21.15.78:0/3136492195 conn(0x7ff1340aa6c0 msgr2=0x7ff1340aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 -- 172.21.15.78:0/3136492195 shutdown_connections 2024-09-15T20:31:51.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 -- 172.21.15.78:0/3136492195 wait complete. 2024-09-15T20:31:51.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 10 monclient: monmap: 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 librados: starting msgr at 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 librados: starting objecter 2024-09-15T20:31:51.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 Processor -- start 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 -- start start 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 librados: setting wanted keys 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 1 librados: calling monclient init 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 10 monclient: init 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.099+0000 7ff13c18c640 10 monclient(hunting): picked mon.noname-a con 0x7ff1340a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient(hunting): picked mon.noname-b con 0x7ff134137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff139f28640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff1340a9610 0x7ff134138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50844/0 (socket says 172.21.15.78:50844) 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient(hunting): picked mon.noname-c con 0x7ff13413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff139f28640 1 -- 172.21.15.78:0/1021121667 learned_addr learned my addr 172.21.15.78:0/1021121667 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 client.?.objecter ms_handle_connect 0x7ff1340a9610 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 client.?.objecter ms_handle_connect 0x7ff13413e5f0 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13413b570 con 0x7ff1340a9610 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff13413c930 con 0x7ff134137ce0 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff134139cd0 con 0x7ff13413e5f0 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102278750 0 0) 0x7ff13413b570 con 0x7ff1340a9610 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient(hunting): authenticate will time out at 1171.988574s 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): my global_id is 4350 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff118004ec0 con 0x7ff1340a9610 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 client.?.objecter ms_handle_connect 0x7ff134137ce0 2024-09-15T20:31:51.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619266972 0 0) 0x7ff134139cd0 con 0x7ff13413e5f0 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): my global_id is 4355 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff13413b570 con 0x7ff13413e5f0 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1731126425 0 0) 0x7ff118004ec0 con 0x7ff1340a9610 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff134139cd0 con 0x7ff1340a9610 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686900219 0 0) 0x7ff13413c930 con 0x7ff134137ce0 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): my global_id is 4354 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1180067b0 con 0x7ff134137ce0 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 348469801 0 0) 0x7ff13413b570 con 0x7ff13413e5f0 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff13413c930 con 0x7ff13413e5f0 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff12c001e20 con 0x7ff13413e5f0 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): dump: 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.115 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1300019a0 con 0x7ff1340a9610 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient(hunting): dump: 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.116 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2477504559 0 0) 0x7ff13413c930 con 0x7ff13413e5f0 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 >> v1:172.21.15.146:6789/0 conn(0x7ff134137ce0 legacy=0x7ff1341395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 >> v1:172.21.15.78:6789/0 conn(0x7ff1340a9610 legacy=0x7ff134138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 monclient: found mon.c 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff13413a5d0 con 0x7ff13413e5f0 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 5 monclient: authenticate success, global_id 4355 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient: _renew_subs 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff12c003c30 con 0x7ff13413e5f0 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff13413c650 con 0x7ff13413e5f0 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff12c003f80 con 0x7ff13413e5f0 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 monclient: dump: 2024-09-15T20:31:51.118 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 client.4355.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient: _renew_subs 2024-09-15T20:31:51.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff13413d280 con 0x7ff13413e5f0 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 1 librados: init done 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12bfff640 4 set_mon_vals no callback set 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff12c00fff0 con 0x7ff13413e5f0 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7ff12c04a2f0 con 0x7ff13413e5f0 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 client.4355.objecter ms_dispatch 0x7ff134000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 3 client.4355.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 3 client.4355.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 20 client.4355.objecter dump_active .. 0 homeless 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff139f28640 10 monclient: get_auth_request con 0x7ff11803e790 auth_method 0 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1340a14b0 con 0x7ff13413e5f0 2024-09-15T20:31:51.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.103+0000 7ff12a7fc640 10 client.4355.objecter ms_handle_connect 0x7ff11803e790 2024-09-15T20:31:51.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.111+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7ff12c015cf0 con 0x7ff13413e5f0 2024-09-15T20:31:51.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.111+0000 7ff12a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.111+0000 7ff12a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.111+0000 7ffbaaf77640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T20:31:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.111+0000 7ffbaaf77640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T20:31:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.111+0000 7ffbaaf77640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.111+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ffb84005380 con 0x7ffba413e5f0 2024-09-15T20:31:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffba17fa640 1 -- 172.21.15.78:0/3766394966 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (unknown 574334944 0 203001719) 0x7ffb9c016660 con 0x7ffba413e5f0 2024-09-15T20:31:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffba17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 0}] 2024-09-15T20:31:51.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffba17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:51.122 INFO:teuthology.orchestra.run.smithi078.stdout:51539607556 2024-09-15T20:31:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 10 librados: watch_flush enter 2024-09-15T20:31:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 10 librados: watch_flush exit 2024-09-15T20:31:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 20 client.4337.objecter shutdown clearing up homeless session... 2024-09-15T20:31:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 10 client.4337.objecter successfully canceled tick 2024-09-15T20:31:51.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 >> v1:172.21.15.146:6812/435598642 conn(0x7ffb8803e300 legacy=0x7ffb880407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 10 monclient: shutdown 2024-09-15T20:31:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 >> v1:172.21.15.78:6790/0 conn(0x7ffba413e5f0 legacy=0x7ffba41d5870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 shutdown_connections 2024-09-15T20:31:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 >> 172.21.15.78:0/3766394966 conn(0x7ffba40aa6c0 msgr2=0x7ffba40a2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 shutdown_connections 2024-09-15T20:31:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 1 -- 172.21.15.78:0/3766394966 wait complete. 2024-09-15T20:31:51.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.115+0000 7ffbaaf77640 1 librados: shutdown 2024-09-15T20:31:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.119+0000 7fd8ef252640 10 monclient: get_monmap_and_config 2024-09-15T20:31:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.119+0000 7fd8ef252640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.119+0000 7fd8ef252640 10 monclient: monmap: 2024-09-15T20:31:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:51.124125+0000 2024-09-15T20:31:51.125 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:51.124125+0000 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.119+0000 7fd8ef252640 1 Processor -- start 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.119+0000 7fd8ef252640 1 -- start start 2024-09-15T20:31:51.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient: init 2024-09-15T20:31:51.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:51.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient(hunting): picked mon.noname-a con 0x7fd8e8137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient(hunting): picked mon.noname-c con 0x7fd8e80a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient(hunting): picked mon.noname-b con 0x7fd8e813e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e80a6420 con 0x7fd8e8137ce0 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e80a6700 con 0x7fd8e80a9610 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e80a2110 con 0x7fd8e813e5f0 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient(hunting): authenticate will time out at 1172.008866s 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ed7ef640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd8e813e5f0 0x7fd8e81409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54706/0 (socket says 172.21.15.78:54706) 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ecfee640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd8e8137ce0 0x7fd8e81380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50852/0 (socket says 172.21.15.78:50852) 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ed7ef640 1 -- 172.21.15.78:0/3211614438 learned_addr learned my addr 172.21.15.78:0/3211614438 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4196782098 0 0) 0x7fd8e80a2110 con 0x7fd8e813e5f0 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient(hunting): my global_id is 4358 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8d8003530 con 0x7fd8e813e5f0 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1544429218 0 0) 0x7fd8e80a6420 con 0x7fd8e8137ce0 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient(hunting): my global_id is 4353 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e80a2110 con 0x7fd8e8137ce0 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3568811675 0 0) 0x7fd8d8003530 con 0x7fd8e813e5f0 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 >> v1:172.21.15.146:6789/0 conn(0x7fd8e80a9610 legacy=0x7fd8e813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 >> v1:172.21.15.78:6789/0 conn(0x7fd8e8137ce0 legacy=0x7fd8e81380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 monclient: found mon.noname-b 2024-09-15T20:31:51.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e80a4bc0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd8e0001a20 con 0x7fd8e813e5f0 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: dump: 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.131 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 5 monclient: authenticate success, global_id 4358 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8e0001bc0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 1 -- 172.21.15.78:0/3211614438 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd8e0001f90 con 0x7fd8e813e5f0 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:51.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8e77fe640 10 monclient: dump: 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 4 set_mon_vals no callback set 2024-09-15T20:31:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 10 monclient: shutdown 2024-09-15T20:31:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 1 -- 172.21.15.78:0/3211614438 >> v1:172.21.15.78:6790/0 conn(0x7fd8e813e5f0 legacy=0x7fd8e81409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 1 -- 172.21.15.78:0/3211614438 shutdown_connections 2024-09-15T20:31:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 1 -- 172.21.15.78:0/3211614438 >> 172.21.15.78:0/3211614438 conn(0x7fd8e80aa6c0 msgr2=0x7fd8e80aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 1 -- 172.21.15.78:0/3211614438 shutdown_connections 2024-09-15T20:31:51.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.123+0000 7fd8ef252640 1 -- 172.21.15.78:0/3211614438 wait complete. 2024-09-15T20:31:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient: monmap: 2024-09-15T20:31:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:51.139 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 librados: starting msgr at 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 librados: starting objecter 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 Processor -- start 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 -- start start 2024-09-15T20:31:51.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 librados: setting wanted keys 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 librados: calling monclient init 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient: init 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient(hunting): picked mon.noname-b con 0x7fd8e80a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient(hunting): picked mon.noname-a con 0x7fd8e8137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient(hunting): picked mon.noname-c con 0x7fd8e813e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e7fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd8e8137ce0 0x7fd8e809d8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50864/0 (socket says 172.21.15.78:50864) 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e7fff640 1 -- 172.21.15.78:0/3988157890 learned_addr learned my addr 172.21.15.78:0/3988157890 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e809f270 con 0x7fd8e8137ce0 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 client.?.objecter ms_handle_connect 0x7fd8e8137ce0 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 client.?.objecter ms_handle_connect 0x7fd8e813e5f0 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e813e3a0 con 0x7fd8e80a9610 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8e8138fe0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 client.?.objecter ms_handle_connect 0x7fd8e80a9610 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient(hunting): authenticate will time out at 1172.014790s 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2807412046 0 0) 0x7fd8e809f270 con 0x7fd8e8137ce0 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): my global_id is 4356 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8cc004ec0 con 0x7fd8e8137ce0 2024-09-15T20:31:51.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1516960406 0 0) 0x7fd8e8138fe0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): my global_id is 4361 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8e809f270 con 0x7fd8e813e5f0 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2921937584 0 0) 0x7fd8e809f270 con 0x7fd8e813e5f0 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8e8138fe0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3772469810 0 0) 0x7fd8cc004ec0 con 0x7fd8e8137ce0 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd8e809f270 con 0x7fd8e8137ce0 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd8e0001d70 con 0x7fd8e813e5f0 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:51.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): dump: 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd8d4001920 con 0x7fd8e8137ce0 2024-09-15T20:31:51.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): dump: 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2821993899 0 0) 0x7fd8e80a2110 con 0x7fd8e80a9610 2024-09-15T20:31:51.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient(hunting): my global_id is 4360 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd8cc004ec0 con 0x7fd8e80a9610 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1851333738 0 0) 0x7fd8e8138fe0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 >> v1:172.21.15.146:6789/0 conn(0x7fd8e80a9610 legacy=0x7fd8e809d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 >> v1:172.21.15.78:6789/0 conn(0x7fd8e8137ce0 legacy=0x7fd8e809d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 monclient: found mon.c 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8e81392c0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 5 monclient: authenticate success, global_id 4361 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd8e0001f10 con 0x7fd8e813e5f0 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient: _renew_subs 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.127+0000 7fd8ef252640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd8e80a0010 con 0x7fd8e813e5f0 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd8e0005120 con 0x7fd8e813e5f0 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 10 monclient: dump: 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.147 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e6ffd640 4 set_mon_vals no callback set 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8ef252640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8ef252640 10 monclient: _renew_subs 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8ef252640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8ef252640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd8e81397c0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8ef252640 1 librados: init done 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd8e0011190 con 0x7fd8e813e5f0 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7fd8e004b710 con 0x7fd8e813e5f0 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 10 client.4361.objecter ms_dispatch 0x7fd8e8000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 3 client.4361.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8d2ffd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 3 client.4361.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8d2ffd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8d2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8d2ffd640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c0005090 con 0x7fd8e813e5f0 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 20 client.4361.objecter dump_active .. 0 homeless 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8ecfee640 10 monclient: get_auth_request con 0x7fd8cc03e210 auth_method 0 2024-09-15T20:31:51.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.131+0000 7fd8e57fa640 10 client.4361.objecter ms_handle_connect 0x7fd8cc03e210 2024-09-15T20:31:51.150 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.0 2024-09-15T20:31:51.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.135+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd8e0016ee0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.135+0000 7fd8e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.135+0000 7fd8e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:51.151 DEBUG:teuthology.parallel:result is None 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.239+0000 7f1e91ffb640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.239+0000 7f1e91ffb640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.239+0000 7f1e91ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.239+0000 7f1e91ffb640 1 -- 172.21.15.78:0/2128361738 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f1e84005380 con 0x7f1eac0a2500 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.239+0000 7f1e93fff640 1 -- 172.21.15.78:0/2128361738 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (unknown 383520633 0 1169801742) 0x7f1ea001ac90 con 0x7f1eac0a2500 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.239+0000 7f1e93fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 3}] 2024-09-15T20:31:51.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.239+0000 7f1e93fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:51.245 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-09-15T20:31:51.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 10 librados: watch_flush enter 2024-09-15T20:31:51.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 10 librados: watch_flush exit 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 20 client.4338.objecter shutdown clearing up homeless session... 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 10 client.4338.objecter successfully canceled tick 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 1 -- 172.21.15.78:0/2128361738 >> v1:172.21.15.146:6812/435598642 conn(0x7f1e9803df30 legacy=0x7f1e980403f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 10 monclient: shutdown 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 1 -- 172.21.15.78:0/2128361738 >> v1:172.21.15.78:6789/0 conn(0x7f1eac0a2500 legacy=0x7f1eac138db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 1 -- 172.21.15.78:0/2128361738 shutdown_connections 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 1 -- 172.21.15.78:0/2128361738 >> 172.21.15.78:0/2128361738 conn(0x7f1eac0a3ce0 msgr2=0x7f1eac071ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 1 -- 172.21.15.78:0/2128361738 shutdown_connections 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 1 -- 172.21.15.78:0/2128361738 wait complete. 2024-09-15T20:31:51.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.243+0000 7f1e91ffb640 1 librados: shutdown 2024-09-15T20:31:51.262 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.3 2024-09-15T20:31:51.262 DEBUG:teuthology.parallel:result is None 2024-09-15T20:31:51.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.295+0000 7f13e6400640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T20:31:51.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.295+0000 7f13e6400640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T20:31:51.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.295+0000 7f13e6400640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.295+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f13e00a3d80 con 0x7f13e013e5f0 2024-09-15T20:31:51.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.295+0000 7f13dd7fa640 1 -- 172.21.15.78:0/2038629713 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (unknown 92182286 0 532354816) 0x7f13d8017040 con 0x7f13e013e5f0 2024-09-15T20:31:51.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.295+0000 7f13dd7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 2}] 2024-09-15T20:31:51.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.295+0000 7f13dd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:51.300 INFO:teuthology.orchestra.run.smithi078.stdout:51539607557 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 10 librados: watch_flush enter 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 10 librados: watch_flush exit 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 20 client.4344.objecter shutdown clearing up homeless session... 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 10 client.4344.objecter successfully canceled tick 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 >> v1:172.21.15.146:6812/435598642 conn(0x7f13c003e0b0 legacy=0x7f13c0040570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 10 monclient: shutdown 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 >> v1:172.21.15.78:6789/0 conn(0x7f13e013e5f0 legacy=0x7f13e01d9230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 shutdown_connections 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 >> 172.21.15.78:0/2038629713 conn(0x7f13e00aa6c0 msgr2=0x7f13e0073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 shutdown_connections 2024-09-15T20:31:51.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 1 -- 172.21.15.78:0/2038629713 wait complete. 2024-09-15T20:31:51.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.299+0000 7f13e6400640 1 librados: shutdown 2024-09-15T20:31:51.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.303+0000 7ff13c18c640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T20:31:51.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.303+0000 7ff13c18c640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T20:31:51.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.303+0000 7ff13c18c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.303+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7ff13413bd00 con 0x7ff13413e5f0 2024-09-15T20:31:51.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.303+0000 7ff12a7fc640 1 -- 172.21.15.78:0/1021121667 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (unknown 1823589948 0 203001719) 0x7ff12c01ef50 con 0x7ff13413e5f0 2024-09-15T20:31:51.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.303+0000 7ff12a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 4}] 2024-09-15T20:31:51.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.303+0000 7ff12a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:51.309 INFO:teuthology.orchestra.run.smithi078.stdout:51539607556 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 10 librados: watch_flush enter 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 10 librados: watch_flush exit 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 20 client.4355.objecter shutdown clearing up homeless session... 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 10 client.4355.objecter successfully canceled tick 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 >> v1:172.21.15.146:6812/435598642 conn(0x7ff11803e790 legacy=0x7ff118040c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 10 monclient: shutdown 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 >> v1:172.21.15.78:6790/0 conn(0x7ff13413e5f0 legacy=0x7ff1341d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 shutdown_connections 2024-09-15T20:31:51.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 >> 172.21.15.78:0/1021121667 conn(0x7ff1340aa6c0 msgr2=0x7ff134073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 shutdown_connections 2024-09-15T20:31:51.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 1 -- 172.21.15.78:0/1021121667 wait complete. 2024-09-15T20:31:51.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.307+0000 7ff13c18c640 1 librados: shutdown 2024-09-15T20:31:51.317 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607557 got 51539607557 for osd.2 2024-09-15T20:31:51.317 DEBUG:teuthology.parallel:result is None 2024-09-15T20:31:51.332 INFO:tasks.ceph.ceph_manager.ceph:need seq 51539607556 got 51539607556 for osd.4 2024-09-15T20:31:51.332 DEBUG:teuthology.parallel:result is None 2024-09-15T20:31:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.331+0000 7fd8d2ffd640 10 start_mon_command cmd=[{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T20:31:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.331+0000 7fd8d2ffd640 10 monclient: _send_command 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T20:31:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.331+0000 7fd8d2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.331+0000 7fd8d2ffd640 1 -- 172.21.15.78:0/3988157890 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7fd8c0005380 con 0x7fd8e813e5f0 2024-09-15T20:31:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8e57fa640 1 -- 172.21.15.78:0/3988157890 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (unknown 832126871 0 1169801742) 0x7fd8e00167e0 con 0x7fd8e813e5f0 2024-09-15T20:31:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd last-stat-seq", "id": 1}] 2024-09-15T20:31:51.337 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:51.338 INFO:teuthology.orchestra.run.smithi078.stdout:47244640261 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 10 librados: watch_flush enter 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 10 librados: watch_flush exit 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 20 client.4361.objecter shutdown clearing up homeless session... 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 10 client.4361.objecter successfully canceled tick 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 1 -- 172.21.15.78:0/3988157890 >> v1:172.21.15.146:6812/435598642 conn(0x7fd8cc03e210 legacy=0x7fd8cc0406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 10 monclient: shutdown 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 1 -- 172.21.15.78:0/3988157890 >> v1:172.21.15.78:6790/0 conn(0x7fd8e813e5f0 legacy=0x7fd8e81d51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 1 -- 172.21.15.78:0/3988157890 shutdown_connections 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 1 -- 172.21.15.78:0/3988157890 >> 172.21.15.78:0/3988157890 conn(0x7fd8e80aa6c0 msgr2=0x7fd8e80a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.340 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 1 -- 172.21.15.78:0/3988157890 shutdown_connections 2024-09-15T20:31:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 1 -- 172.21.15.78:0/3988157890 wait complete. 2024-09-15T20:31:51.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.335+0000 7fd8d2ffd640 1 librados: shutdown 2024-09-15T20:31:51.354 INFO:tasks.ceph.ceph_manager.ceph:need seq 47244640261 got 47244640261 for osd.1 2024-09-15T20:31:51.354 DEBUG:teuthology.parallel:result is None 2024-09-15T20:31:51.354 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T20:31:51.354 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.495+0000 7f08b7b86640 10 monclient: get_monmap_and_config 2024-09-15T20:31:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.495+0000 7f08b7b86640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.495+0000 7f08b7b86640 10 monclient: monmap: 2024-09-15T20:31:51.500 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:51.499773+0000 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:51.499773+0000 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:51.501 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.495+0000 7f08b7b86640 1 Processor -- start 2024-09-15T20:31:51.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.495+0000 7f08b7b86640 1 -- start start 2024-09-15T20:31:51.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.495+0000 7f08b7b86640 10 monclient: init 2024-09-15T20:31:51.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): picked mon.noname-b con 0x7f08b0074630 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): picked mon.noname-a con 0x7f08b009d950 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): picked mon.noname-c con 0x7f08b0074a10 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08b0134bc0 con 0x7f08b009d950 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08b009f900 con 0x7f08b0074a10 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08b009fbe0 con 0x7f08b0074630 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): authenticate will time out at 1172.384437s 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b5121640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f08b009d950 0x7f08b01344b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50878/0 (socket says 172.21.15.78:50878) 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b5121640 1 -- 172.21.15.78:0/2352120102 learned_addr learned my addr 172.21.15.78:0/2352120102 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2068768026 0 0) 0x7f08b0134bc0 con 0x7f08b009d950 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient(hunting): my global_id is 4359 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f089c003530 con 0x7f08b009d950 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1106727778 0 0) 0x7f08b009fbe0 con 0x7f08b0074630 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient(hunting): my global_id is 4364 2024-09-15T20:31:51.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08b0134bc0 con 0x7f08b0074630 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1322034834 0 0) 0x7f089c003530 con 0x7f08b009d950 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 >> v1:172.21.15.78:6790/0 conn(0x7f08b0074630 legacy=0x7f08b0133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 >> v1:172.21.15.146:6789/0 conn(0x7f08b0074a10 legacy=0x7f08b013d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 monclient: found mon.noname-a 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b009fec0 con 0x7f08b009d950 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f08a00015d0 con 0x7f08b009d950 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 5 monclient: authenticate success, global_id 4359 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: dump: 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.513 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f08a0001770 con 0x7f08b009d950 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 1 -- 172.21.15.78:0/2352120102 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f08a0001b40 con 0x7f08b009d950 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b4920640 10 monclient: dump: 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.514 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 4 set_mon_vals no callback set 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: shutdown 2024-09-15T20:31:51.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- 172.21.15.78:0/2352120102 >> v1:172.21.15.78:6789/0 conn(0x7f08b009d950 legacy=0x7f08b01344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- 172.21.15.78:0/2352120102 shutdown_connections 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- 172.21.15.78:0/2352120102 >> 172.21.15.78:0/2352120102 conn(0x7f08b00a63c0 msgr2=0x7f08b00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- 172.21.15.78:0/2352120102 shutdown_connections 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- 172.21.15.78:0/2352120102 wait complete. 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: monmap: 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.516 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 librados: starting msgr at 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 librados: starting objecter 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 Processor -- start 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- start start 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 librados: setting wanted keys 2024-09-15T20:31:51.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 librados: calling monclient init 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: init 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): picked mon.noname-a con 0x7f08b0074630 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): picked mon.noname-b con 0x7f08b0074a10 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): picked mon.noname-c con 0x7f08b009d950 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08b011d7b0 con 0x7f08b0074630 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08b00741e0 con 0x7f08b0074a10 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08b011e720 con 0x7f08b009d950 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b7b86640 10 monclient(hunting): authenticate will time out at 1172.387423s 2024-09-15T20:31:51.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b6123640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f08b009d950 0x7f08b01d4ba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54734/0 (socket says 172.21.15.78:54734) 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b5922640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f08b0074630 0x7f08b012cd00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50892/0 (socket says 172.21.15.78:50892) 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08b6123640 1 -- 172.21.15.78:0/2580128327 learned_addr learned my addr 172.21.15.78:0/2580128327 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 client.?.objecter ms_handle_connect 0x7f08b009d950 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 client.?.objecter ms_handle_connect 0x7f08b0074630 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1974694707 0 0) 0x7f08b011e720 con 0x7f08b009d950 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 monclient(hunting): my global_id is 4367 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0894004ec0 con 0x7f08b009d950 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3858752541 0 0) 0x7f08b011d7b0 con 0x7f08b0074630 2024-09-15T20:31:51.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 monclient(hunting): my global_id is 4362 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08b011e720 con 0x7f08b0074630 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 client.?.objecter ms_handle_connect 0x7f08b0074a10 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.499+0000 7f08a67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3530935019 0 0) 0x7f0894004ec0 con 0x7f08b009d950 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f08b011d7b0 con 0x7f08b009d950 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f08a8001d40 con 0x7f08b009d950 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): dump: 2024-09-15T20:31:51.520 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 44431456 0 0) 0x7f08b011e720 con 0x7f08b0074630 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0894004ec0 con 0x7f08b0074630 2024-09-15T20:31:51.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f08ac001940 con 0x7f08b0074630 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): dump: 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1632583625 0 0) 0x7f08b00741e0 con 0x7f08b0074a10 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient(hunting): my global_id is 4366 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08b011e720 con 0x7f08b0074a10 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1116283413 0 0) 0x7f08b011d7b0 con 0x7f08b009d950 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 >> v1:172.21.15.146:6789/0 conn(0x7f08b0074a10 legacy=0x7f08b01cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 >> v1:172.21.15.78:6789/0 conn(0x7f08b0074630 legacy=0x7f08b012cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 monclient: found mon.c 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b011de80 con 0x7f08b009d950 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 5 monclient: authenticate success, global_id 4367 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 10 monclient: _renew_subs 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f08b012d410 con 0x7f08b009d950 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 10 monclient: _renew_subs 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f08b01cd0e0 con 0x7f08b009d950 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 1 librados: init done 2024-09-15T20:31:51.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f08a8002040 con 0x7f08b009d950 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a7fff640 4 set_mon_vals no callback set 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f08a8003df0 con 0x7f08b009d950 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 monclient: dump: 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.525 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0898005090 con 0x7f08b009d950 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f08a8003f90 con 0x7f08b009d950 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f08a804b360 con 0x7f08b009d950 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 client.4367.objecter ms_dispatch 0x7f08b0000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 3 client.4367.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 3 client.4367.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:51.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 20 client.4367.objecter dump_active .. 0 homeless 2024-09-15T20:31:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08b5922640 10 monclient: get_auth_request con 0x7f089403e180 auth_method 0 2024-09-15T20:31:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.503+0000 7f08a67fc640 10 client.4367.objecter ms_handle_connect 0x7f089403e180 2024-09-15T20:31:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.507+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f08a8016e60 con 0x7f08b009d950 2024-09-15T20:31:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.507+0000 7f08a67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.507+0000 7f08a67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:51.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.659+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0898000ba0 con 0x7f089403e180 2024-09-15T20:31:51.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.659+0000 7f08a67fc640 1 -- 172.21.15.78:0/2580128327 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34203 (unknown 2965378022 0 4172207700) 0x7f0898000ba0 con 0x7f089403e180 2024-09-15T20:31:51.665 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:51.666 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 10 librados: watch_flush enter 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 10 librados: watch_flush exit 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 20 client.4367.objecter shutdown clearing up homeless session... 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 10 client.4367.objecter successfully canceled tick 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 >> v1:172.21.15.146:6812/435598642 conn(0x7f089403e180 legacy=0x7f0894040640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 10 monclient: shutdown 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 >> v1:172.21.15.78:6790/0 conn(0x7f08b009d950 legacy=0x7f08b01d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 shutdown_connections 2024-09-15T20:31:51.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 >> 172.21.15.78:0/2580128327 conn(0x7f08b00a63c0 msgr2=0x7f08b0132540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 shutdown_connections 2024-09-15T20:31:51.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 1 -- 172.21.15.78:0/2580128327 wait complete. 2024-09-15T20:31:51.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.663+0000 7f08b7b86640 1 librados: shutdown 2024-09-15T20:31:51.681 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-15T20:31:50.654280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163272,"kb_used_data":1088,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067768,"statfs":{"total":579820584960,"available":579653394432,"internally_reserved":0,"allocated":1114112,"data_stored":1371166,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.872055"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825675+0000","last_change":"2024-09-15T20:31:47.826026+0000","last_active":"2024-09-15T20:31:47.825675+0000","last_peered":"2024-09-15T20:31:47.825675+0000","last_clean":"2024-09-15T20:31:47.825675+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:31:47.825675+0000","last_undegraded":"2024-09-15T20:31:47.825675+0000","last_fullsized":"2024-09-15T20:31:47.825675+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:19.315897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837983+0000","last_change":"2024-09-15T20:31:47.838456+0000","last_active":"2024-09-15T20:31:47.837983+0000","last_peered":"2024-09-15T20:31:47.837983+0000","last_clean":"2024-09-15T20:31:47.837983+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:31:47.837983+0000","last_undegraded":"2024-09-15T20:31:47.837983+0000","last_fullsized":"2024-09-15T20:31:47.837983+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:31:54.370290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825214+0000","last_change":"2024-09-15T20:31:47.825665+0000","last_active":"2024-09-15T20:31:47.825214+0000","last_peered":"2024-09-15T20:31:47.825214+0000","last_clean":"2024-09-15T20:31:47.825214+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:31:47.825214+0000","last_undegraded":"2024-09-15T20:31:47.825214+0000","last_fullsized":"2024-09-15T20:31:47.825214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:40:22.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837836+0000","last_change":"2024-09-15T20:31:47.838237+0000","last_active":"2024-09-15T20:31:47.837836+0000","last_peered":"2024-09-15T20:31:47.837836+0000","last_clean":"2024-09-15T20:31:47.837836+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:31:47.837836+0000","last_undegraded":"2024-09-15T20:31:47.837836+0000","last_fullsized":"2024-09-15T20:31:47.837836+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:49.045779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837218+0000","last_change":"2024-09-15T20:31:47.834864+0000","last_active":"2024-09-15T20:31:47.837218+0000","last_peered":"2024-09-15T20:31:47.837218+0000","last_clean":"2024-09-15T20:31:47.837218+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:31:47.837218+0000","last_undegraded":"2024-09-15T20:31:47.837218+0000","last_fullsized":"2024-09-15T20:31:47.837218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:56.096771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.969706+0000","last_change":"2024-09-15T20:31:47.969706+0000","last_active":"2024-09-15T20:31:47.969706+0000","last_peered":"2024-09-15T20:31:47.969706+0000","last_clean":"2024-09-15T20:31:47.969706+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:31:47.969706+0000","last_undegraded":"2024-09-15T20:31:47.969706+0000","last_fullsized":"2024-09-15T20:31:47.969706+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:48.645009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825141+0000","last_change":"2024-09-15T20:31:47.825565+0000","last_active":"2024-09-15T20:31:47.825141+0000","last_peered":"2024-09-15T20:31:47.825141+0000","last_clean":"2024-09-15T20:31:47.825141+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:31:47.825141+0000","last_undegraded":"2024-09-15T20:31:47.825141+0000","last_fullsized":"2024-09-15T20:31:47.825141+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:49:12.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.833645+0000","last_change":"2024-09-15T20:31:47.833645+0000","last_active":"2024-09-15T20:31:47.833645+0000","last_peered":"2024-09-15T20:31:47.833645+0000","last_clean":"2024-09-15T20:31:47.833645+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:31:47.833645+0000","last_undegraded":"2024-09-15T20:31:47.833645+0000","last_fullsized":"2024-09-15T20:31:47.833645+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:24.343248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.824491+0000","last_change":"2024-09-15T20:31:40.658325+0000","last_active":"2024-09-15T20:31:47.824491+0000","last_peered":"2024-09-15T20:31:47.824491+0000","last_clean":"2024-09-15T20:31:47.824491+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:31:47.824491+0000","last_undegraded":"2024-09-15T20:31:47.824491+0000","last_fullsized":"2024-09-15T20:31:47.824491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_clean_scrub_stamp":"2024-09-15T20:31:38.726961+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:41.743437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":52212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":581140,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":581140,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:51.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient: get_monmap_and_config 2024-09-15T20:31:51.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient: monmap: 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:51.787066+0000 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:51.787066+0000 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:51.788 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 1 Processor -- start 2024-09-15T20:31:51.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 1 -- start start 2024-09-15T20:31:51.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient: init 2024-09-15T20:31:51.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient(hunting): picked mon.noname-c con 0x7f6828137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient(hunting): picked mon.noname-a con 0x7f6828074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient(hunting): picked mon.noname-b con 0x7f682813e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6828138850 con 0x7f6828074500 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f68280a5da0 con 0x7f6828137d60 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6828073560 con 0x7f682813e370 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682fb1e640 10 monclient(hunting): authenticate will time out at 1172.671731s 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682e0bb640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f682813e370 0x7f6828140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54746/0 (socket says 172.21.15.78:54746) 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682d0b9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6828074500 0x7f682813cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50894/0 (socket says 172.21.15.78:50894) 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.783+0000 7f682e0bb640 1 -- 172.21.15.78:0/3859650572 learned_addr learned my addr 172.21.15.78:0/3859650572 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4229001350 0 0) 0x7f6828073560 con 0x7f682813e370 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient(hunting): my global_id is 4370 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6814003530 con 0x7f682813e370 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1614959068 0 0) 0x7f6828138850 con 0x7f6828074500 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient(hunting): my global_id is 4365 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6828073560 con 0x7f6828074500 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3233979729 0 0) 0x7f6814003530 con 0x7f682813e370 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 >> v1:172.21.15.146:6789/0 conn(0x7f6828137d60 legacy=0x7f6828138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 >> v1:172.21.15.78:6789/0 conn(0x7f6828074500 legacy=0x7f682813cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 monclient: found mon.noname-b 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6828073840 con 0x7f682813e370 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6820001a20 con 0x7f682813e370 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: dump: 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.795 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 5 monclient: authenticate success, global_id 4370 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6820001c40 con 0x7f682813e370 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 1 -- 172.21.15.78:0/3859650572 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6820002050 con 0x7f682813e370 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 4 set_mon_vals no callback set 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682c8b8640 10 monclient: dump: 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.796 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: shutdown 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- 172.21.15.78:0/3859650572 >> v1:172.21.15.78:6790/0 conn(0x7f682813e370 legacy=0x7f6828140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- 172.21.15.78:0/3859650572 shutdown_connections 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- 172.21.15.78:0/3859650572 >> 172.21.15.78:0/3859650572 conn(0x7f68280aa650 msgr2=0x7f68280aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- 172.21.15.78:0/3859650572 shutdown_connections 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- 172.21.15.78:0/3859650572 wait complete. 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: build_initial_monmap 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: monmap: 2024-09-15T20:31:51.797 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 librados: starting msgr at 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 librados: starting objecter 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 Processor -- start 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- start start 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 librados: setting wanted keys 2024-09-15T20:31:51.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 librados: calling monclient init 2024-09-15T20:31:51.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: init 2024-09-15T20:31:51.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:51.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:51.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient(hunting): picked mon.noname-b con 0x7f6828074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:51.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient(hunting): picked mon.noname-c con 0x7f6828137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient(hunting): picked mon.noname-a con 0x7f682813e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682813c880 con 0x7f682813e370 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6828139880 con 0x7f6828074500 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f682813ceb0 con 0x7f6828137d60 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient(hunting): authenticate will time out at 1172.674800s 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682e0bb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f682813e370 0x7f68281d5010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50904/0 (socket says 172.21.15.78:50904) 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682e0bb640 1 -- 172.21.15.78:0/3756997579 learned_addr learned my addr 172.21.15.78:0/3756997579 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682d0b9640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f6828137d60 0x7f6828139170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54754/0 (socket says 172.21.15.78:54754) 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 client.?.objecter ms_handle_connect 0x7f682813e370 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 client.?.objecter ms_handle_connect 0x7f6828137d60 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1949829469 0 0) 0x7f682813c880 con 0x7f682813e370 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): my global_id is 4368 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f680c004ec0 con 0x7f682813e370 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1231304739 0 0) 0x7f682813ceb0 con 0x7f6828137d60 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): my global_id is 4373 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f682813c880 con 0x7f6828137d60 2024-09-15T20:31:51.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 client.?.objecter ms_handle_connect 0x7f6828074500 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1005691790 0 0) 0x7f680c004ec0 con 0x7f682813e370 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f682813ceb0 con 0x7f682813e370 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6820001f90 con 0x7f682813e370 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): dump: 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.805 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 345975717 0 0) 0x7f682813c880 con 0x7f6828137d60 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f680c004ec0 con 0x7f6828137d60 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6818001940 con 0x7f6828137d60 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): dump: 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.806 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3479511130 0 0) 0x7f6828139880 con 0x7f6828074500 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient(hunting): my global_id is 4372 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f682813c880 con 0x7f6828074500 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1993049672 0 0) 0x7f682813ceb0 con 0x7f682813e370 2024-09-15T20:31:51.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 >> v1:172.21.15.78:6790/0 conn(0x7f6828137d60 legacy=0x7f6828139170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 >> v1:172.21.15.146:6789/0 conn(0x7f6828074500 legacy=0x7f6828138a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 monclient: found mon.a 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68280a6530 con 0x7f682813e370 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:51.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 5 monclient: authenticate success, global_id 4368 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: _renew_subs 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f682813a520 con 0x7f682813e370 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 client.4368.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: _renew_subs 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f682813b080 con 0x7f682813e370 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f682fb1e640 1 librados: init done 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6820001510 con 0x7f682813e370 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6820004f80 con 0x7f682813e370 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681ffff640 4 set_mon_vals no callback set 2024-09-15T20:31:51.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.787+0000 7f681e7fc640 10 monclient: dump: 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f682fb1e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f682fb1e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f682fb1e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6810005090 con 0x7f682813e370 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f6820010ff0 con 0x7f682813e370 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f682004b360 con 0x7f682813e370 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f681e7fc640 10 client.4368.objecter ms_dispatch 0x7f6828000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f681e7fc640 3 client.4368.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f681e7fc640 3 client.4368.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f681e7fc640 20 client.4368.objecter dump_active .. 0 homeless 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f682d8ba640 10 monclient: get_auth_request con 0x7f680c03e2c0 auth_method 0 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.791+0000 7f681e7fc640 10 client.4368.objecter ms_handle_connect 0x7f680c03e2c0 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.795+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f6820016da0 con 0x7f682813e370 2024-09-15T20:31:51.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.795+0000 7f681e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:51.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.795+0000 7f681e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:51.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.951+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6810000ba0 con 0x7f680c03e2c0 2024-09-15T20:31:51.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.951+0000 7f681e7fc640 1 -- 172.21.15.78:0/3756997579 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34203 (unknown 2965378022 0 4172207700) 0x7f6810000ba0 con 0x7f680c03e2c0 2024-09-15T20:31:51.957 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:51.957 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 10 librados: watch_flush enter 2024-09-15T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 10 librados: watch_flush exit 2024-09-15T20:31:51.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 20 client.4368.objecter shutdown clearing up homeless session... 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 10 client.4368.objecter successfully canceled tick 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 >> v1:172.21.15.146:6812/435598642 conn(0x7f680c03e2c0 legacy=0x7f680c040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 10 monclient: shutdown 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 >> v1:172.21.15.78:6789/0 conn(0x7f682813e370 legacy=0x7f68281d5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 shutdown_connections 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 >> 172.21.15.78:0/3756997579 conn(0x7f68280aa650 msgr2=0x7f68280a8d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 shutdown_connections 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 1 -- 172.21.15.78:0/3756997579 wait complete. 2024-09-15T20:31:51.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:51.955+0000 7f682fb1e640 1 librados: shutdown 2024-09-15T20:31:51.973 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":21,"stamp":"2024-09-15T20:31:50.654280+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163272,"kb_used_data":1088,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067768,"statfs":{"total":579820584960,"available":579653394432,"internally_reserved":0,"allocated":1114112,"data_stored":1371166,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.872055"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825675+0000","last_change":"2024-09-15T20:31:47.826026+0000","last_active":"2024-09-15T20:31:47.825675+0000","last_peered":"2024-09-15T20:31:47.825675+0000","last_clean":"2024-09-15T20:31:47.825675+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:31:47.825675+0000","last_undegraded":"2024-09-15T20:31:47.825675+0000","last_fullsized":"2024-09-15T20:31:47.825675+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:19.315897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837983+0000","last_change":"2024-09-15T20:31:47.838456+0000","last_active":"2024-09-15T20:31:47.837983+0000","last_peered":"2024-09-15T20:31:47.837983+0000","last_clean":"2024-09-15T20:31:47.837983+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:31:47.837983+0000","last_undegraded":"2024-09-15T20:31:47.837983+0000","last_fullsized":"2024-09-15T20:31:47.837983+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:31:54.370290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825214+0000","last_change":"2024-09-15T20:31:47.825665+0000","last_active":"2024-09-15T20:31:47.825214+0000","last_peered":"2024-09-15T20:31:47.825214+0000","last_clean":"2024-09-15T20:31:47.825214+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:31:47.825214+0000","last_undegraded":"2024-09-15T20:31:47.825214+0000","last_fullsized":"2024-09-15T20:31:47.825214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:40:22.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837836+0000","last_change":"2024-09-15T20:31:47.838237+0000","last_active":"2024-09-15T20:31:47.837836+0000","last_peered":"2024-09-15T20:31:47.837836+0000","last_clean":"2024-09-15T20:31:47.837836+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:31:47.837836+0000","last_undegraded":"2024-09-15T20:31:47.837836+0000","last_fullsized":"2024-09-15T20:31:47.837836+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:49.045779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837218+0000","last_change":"2024-09-15T20:31:47.834864+0000","last_active":"2024-09-15T20:31:47.837218+0000","last_peered":"2024-09-15T20:31:47.837218+0000","last_clean":"2024-09-15T20:31:47.837218+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:31:47.837218+0000","last_undegraded":"2024-09-15T20:31:47.837218+0000","last_fullsized":"2024-09-15T20:31:47.837218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:56.096771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.969706+0000","last_change":"2024-09-15T20:31:47.969706+0000","last_active":"2024-09-15T20:31:47.969706+0000","last_peered":"2024-09-15T20:31:47.969706+0000","last_clean":"2024-09-15T20:31:47.969706+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:31:47.969706+0000","last_undegraded":"2024-09-15T20:31:47.969706+0000","last_fullsized":"2024-09-15T20:31:47.969706+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:48.645009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825141+0000","last_change":"2024-09-15T20:31:47.825565+0000","last_active":"2024-09-15T20:31:47.825141+0000","last_peered":"2024-09-15T20:31:47.825141+0000","last_clean":"2024-09-15T20:31:47.825141+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:31:47.825141+0000","last_undegraded":"2024-09-15T20:31:47.825141+0000","last_fullsized":"2024-09-15T20:31:47.825141+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:49:12.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.833645+0000","last_change":"2024-09-15T20:31:47.833645+0000","last_active":"2024-09-15T20:31:47.833645+0000","last_peered":"2024-09-15T20:31:47.833645+0000","last_clean":"2024-09-15T20:31:47.833645+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:31:47.833645+0000","last_undegraded":"2024-09-15T20:31:47.833645+0000","last_fullsized":"2024-09-15T20:31:47.833645+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:24.343248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.824491+0000","last_change":"2024-09-15T20:31:40.658325+0000","last_active":"2024-09-15T20:31:47.824491+0000","last_peered":"2024-09-15T20:31:47.824491+0000","last_clean":"2024-09-15T20:31:47.824491+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:31:47.824491+0000","last_undegraded":"2024-09-15T20:31:47.824491+0000","last_fullsized":"2024-09-15T20:31:47.824491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_clean_scrub_stamp":"2024-09-15T20:31:38.726961+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:41.743437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":52212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":581140,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":581140,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:51.974 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T20:31:51.974 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-15T20:31:51.974 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-09-15T20:31:51.975 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-09-15T20:31:52.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient: get_monmap_and_config 2024-09-15T20:31:52.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient: build_initial_monmap 2024-09-15T20:31:52.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient: monmap: 2024-09-15T20:31:52.078 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:52.078 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.078 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:52.077670+0000 2024-09-15T20:31:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:52.077670+0000 2024-09-15T20:31:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:52.079 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:52.079 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 1 Processor -- start 2024-09-15T20:31:52.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 1 -- start start 2024-09-15T20:31:52.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient: init 2024-09-15T20:31:52.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:52.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:52.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2aa80a0ff0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:52.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2aa8073800 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2aa80a1ae0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa8072df0 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa8074110 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa80743f0 con 0x7f2aa8073800 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient(hunting): authenticate will time out at 1172.962646s 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aacd95640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f2aa8073800 0x7f2aa813ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54756/0 (socket says 172.21.15.78:54756) 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aacd95640 1 -- 172.21.15.78:0/3864774849 learned_addr learned my addr 172.21.15.78:0/3864774849 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4093107410 0 0) 0x7f2aa8072df0 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient(hunting): my global_id is 4371 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a94003530 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3642518673 0 0) 0x7f2aa80743f0 con 0x7f2aa8073800 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient(hunting): my global_id is 4376 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa8072df0 con 0x7f2aa8073800 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3918675672 0 0) 0x7f2a94003530 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 >> v1:172.21.15.78:6790/0 conn(0x7f2aa8073800 legacy=0x7f2aa813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 >> v1:172.21.15.146:6789/0 conn(0x7f2aa80a0ff0 legacy=0x7f2aa80a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 monclient: found mon.noname-a 2024-09-15T20:31:52.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa8074eb0 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: _finish_auth 0 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 5 monclient: authenticate success, global_id 4371 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2aa0001950 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:52.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: dump: 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2aa0001af0 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:52.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 1 -- 172.21.15.78:0/3864774849 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2aa0001f00 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 4 set_mon_vals no callback set 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2a9ffff640 10 monclient: dump: 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.087 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 10 monclient: shutdown 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.075+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/3864774849 >> v1:172.21.15.78:6789/0 conn(0x7f2aa80a1ae0 legacy=0x7f2aa8140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/3864774849 shutdown_connections 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/3864774849 >> 172.21.15.78:0/3864774849 conn(0x7f2aa80a63c0 msgr2=0x7f2aa80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/3864774849 shutdown_connections 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/3864774849 wait complete. 2024-09-15T20:31:52.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: build_initial_monmap 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: monmap: 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 librados: starting msgr at 2024-09-15T20:31:52.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 librados: starting objecter 2024-09-15T20:31:52.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 Processor -- start 2024-09-15T20:31:52.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- start start 2024-09-15T20:31:52.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 librados: setting wanted keys 2024-09-15T20:31:52.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 librados: calling monclient init 2024-09-15T20:31:52.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: init 2024-09-15T20:31:52.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:52.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:52.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient(hunting): picked mon.noname-a con 0x7f2aa8073800 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:52.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient(hunting): picked mon.noname-c con 0x7f2aa80a0ff0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:52.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient(hunting): picked mon.noname-b con 0x7f2aa80a1ae0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:52.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa813aa10 con 0x7f2aa8073800 2024-09-15T20:31:52.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa813acf0 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2aa812e790 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:52.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient(hunting): authenticate will time out at 1172.965629s 2024-09-15T20:31:52.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aacd95640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f2aa80a0ff0 0x7f2aa8137540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54758/0 (socket says 172.21.15.78:54758) 2024-09-15T20:31:52.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aad596640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2aa8073800 0x7f2aa8136e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50932/0 (socket says 172.21.15.78:50932) 2024-09-15T20:31:52.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aacd95640 1 -- 172.21.15.78:0/1408375738 learned_addr learned my addr 172.21.15.78:0/1408375738 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:52.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 client.?.objecter ms_handle_connect 0x7f2aa80a0ff0 2024-09-15T20:31:52.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 client.?.objecter ms_handle_connect 0x7f2aa8073800 2024-09-15T20:31:52.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2039010588 0 0) 0x7f2aa812e790 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): my global_id is 4379 2024-09-15T20:31:52.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2a8c004ec0 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2589326769 0 0) 0x7f2aa813aa10 con 0x7f2aa8073800 2024-09-15T20:31:52.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): my global_id is 4374 2024-09-15T20:31:52.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa812e790 con 0x7f2aa8073800 2024-09-15T20:31:52.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 client.?.objecter ms_handle_connect 0x7f2aa80a1ae0 2024-09-15T20:31:52.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1151123370 0 0) 0x7f2a8c004ec0 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2aa813aa10 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2a98001300 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:52.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:52.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): dump: 2024-09-15T20:31:52.102 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.102 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.102 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.102 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.102 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.103 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.103 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.103 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.103 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.103 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:52.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 332115238 0 0) 0x7f2aa812e790 con 0x7f2aa8073800 2024-09-15T20:31:52.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2a8c004ec0 con 0x7f2aa8073800 2024-09-15T20:31:52.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2aa4001940 con 0x7f2aa8073800 2024-09-15T20:31:52.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:52.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:52.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): dump: 2024-09-15T20:31:52.105 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.105 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.105 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.105 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.106 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.106 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.106 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.106 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.106 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.107 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1709104925 0 0) 0x7f2aa813acf0 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient(hunting): my global_id is 4378 2024-09-15T20:31:52.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2aa812e790 con 0x7f2aa80a1ae0 2024-09-15T20:31:52.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 800440868 0 0) 0x7f2aa813aa10 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 >> v1:172.21.15.146:6789/0 conn(0x7f2aa80a1ae0 legacy=0x7f2aa8137c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 >> v1:172.21.15.78:6789/0 conn(0x7f2aa8073800 legacy=0x7f2aa8136e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:52.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 monclient: found mon.c 2024-09-15T20:31:52.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:52.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2aa809da20 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:52.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:52.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:52.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 5 monclient: authenticate success, global_id 4379 2024-09-15T20:31:52.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: _renew_subs 2024-09-15T20:31:52.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2aa809d180 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:52.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: _renew_subs 2024-09-15T20:31:52.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2aa8138dd0 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 librados: init done 2024-09-15T20:31:52.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2a980047f0 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:52.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2a98004b40 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:52.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9f7fe640 4 set_mon_vals no callback set 2024-09-15T20:31:52.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 monclient: dump: 2024-09-15T20:31:52.113 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.113 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.113 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.114 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.114 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.114 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.114 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.114 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.114 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.115 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a88005090 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f2a98010bb0 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f2a9804b0f0 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 10 client.4379.objecter ms_dispatch 0x7f2aa8000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:52.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 3 client.4379.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:52.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 3 client.4379.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:52.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2a9dffb640 20 client.4379.objecter dump_active .. 0 homeless 2024-09-15T20:31:52.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.079+0000 7f2aad596640 10 monclient: get_auth_request con 0x7f2a8c03e160 auth_method 0 2024-09-15T20:31:52.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.083+0000 7f2a9dffb640 10 client.4379.objecter ms_handle_connect 0x7f2a8c03e160 2024-09-15T20:31:52.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.083+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f2a98016b50 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.083+0000 7f2a9dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.083+0000 7f2a9dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:52.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.275+0000 7f2aaf7fa640 10 start_mon_command cmd=[{"prefix": "health", "format": "json"}] 2024-09-15T20:31:52.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.275+0000 7f2aaf7fa640 10 monclient: _send_command 2 [{"prefix": "health", "format": "json"}] 2024-09-15T20:31:52.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.275+0000 7f2aaf7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.275+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2a88005380 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.275+0000 7f2a9dffb640 1 -- 172.21.15.78:0/1408375738 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (unknown 3730786198 0 4185958460) 0x7f2a98016450 con 0x7f2aa80a0ff0 2024-09-15T20:31:52.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.275+0000 7f2a9dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "health", "format": "json"}] 2024-09-15T20:31:52.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.275+0000 7f2a9dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:52.281 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:52.281 INFO:teuthology.orchestra.run.smithi078.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-15T20:31:52.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 10 librados: watch_flush enter 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 10 librados: watch_flush exit 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 20 client.4379.objecter shutdown clearing up homeless session... 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 10 client.4379.objecter successfully canceled tick 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 >> v1:172.21.15.146:6812/435598642 conn(0x7f2a8c03e160 legacy=0x7f2a8c040620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 10 monclient: shutdown 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 >> v1:172.21.15.78:6790/0 conn(0x7f2aa80a0ff0 legacy=0x7f2aa8137540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 shutdown_connections 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 >> 172.21.15.78:0/1408375738 conn(0x7f2aa80a63c0 msgr2=0x7f2aa81309d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 shutdown_connections 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 1 -- 172.21.15.78:0/1408375738 wait complete. 2024-09-15T20:31:52.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.279+0000 7f2aaf7fa640 1 librados: shutdown 2024-09-15T20:31:52.297 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-09-15T20:31:52.297 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-09-15T20:31:52.310 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-09-15T20:31:52.311 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient: get_monmap_and_config 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient: build_initial_monmap 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient: monmap: 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:52.422315+0000 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:52.422315+0000 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.423 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:52.424 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:52.424 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 1 Processor -- start 2024-09-15T20:31:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 1 -- start start 2024-09-15T20:31:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient: init 2024-09-15T20:31:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:31:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient(hunting): picked mon.noname-c con 0x7f65e0135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient(hunting): picked mon.noname-b con 0x7f65e00a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:52.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient(hunting): picked mon.noname-a con 0x7f65e013c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e00745e0 con 0x7f65e013c220 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e006ecf0 con 0x7f65e0135b60 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e006efd0 con 0x7f65e00a24e0 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient(hunting): authenticate will time out at 1173.306856s 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65df59e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f65e013c220 0x7f65e013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50946/0 (socket says 172.21.15.78:50946) 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65df59e640 1 -- 172.21.15.78:0/4082874032 learned_addr learned my addr 172.21.15.78:0/4082874032 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3387564913 0 0) 0x7f65e006efd0 con 0x7f65e00a24e0 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): my global_id is 4382 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65cc003530 con 0x7f65e00a24e0 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4150357004 0 0) 0x7f65e00745e0 con 0x7f65e013c220 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): my global_id is 4377 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e006efd0 con 0x7f65e013c220 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655537862 0 0) 0x7f65e006ecf0 con 0x7f65e0135b60 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient(hunting): my global_id is 4381 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e00745e0 con 0x7f65e0135b60 2024-09-15T20:31:52.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2507442963 0 0) 0x7f65e006efd0 con 0x7f65e013c220 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 >> v1:172.21.15.78:6790/0 conn(0x7f65e00a24e0 legacy=0x7f65e013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 >> v1:172.21.15.146:6789/0 conn(0x7f65e0135b60 legacy=0x7f65e0135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: _finish_hunting 0 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 monclient: found mon.noname-a 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e0136650 con 0x7f65e013c220 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: _finish_auth 0 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: _check_auth_tickets 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65d8001980 con 0x7f65e013c220 2024-09-15T20:31:52.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: dump: 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 5 monclient: authenticate success, global_id 4377 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:52.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65d8001b20 con 0x7f65e013c220 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 1 -- 172.21.15.78:0/4082874032 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65d8001f30 con 0x7f65e013c220 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65de59c640 10 monclient: dump: 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.435 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.419+0000 7f65e51de640 4 set_mon_vals no callback set 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: shutdown 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- 172.21.15.78:0/4082874032 >> v1:172.21.15.78:6789/0 conn(0x7f65e013c220 legacy=0x7f65e013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- 172.21.15.78:0/4082874032 shutdown_connections 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- 172.21.15.78:0/4082874032 >> 172.21.15.78:0/4082874032 conn(0x7f65e00a63c0 msgr2=0x7f65e00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- 172.21.15.78:0/4082874032 shutdown_connections 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- 172.21.15.78:0/4082874032 wait complete. 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: build_initial_monmap 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: monmap: 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:52.436 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 librados: starting msgr at 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 librados: starting objecter 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 Processor -- start 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- start start 2024-09-15T20:31:52.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 librados: setting wanted keys 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 librados: calling monclient init 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: init 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient(hunting): picked mon.noname-a con 0x7f65e00a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient(hunting): picked mon.noname-c con 0x7f65e0135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient(hunting): picked mon.noname-b con 0x7f65e013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e009d180 con 0x7f65e00a24e0 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e0138320 con 0x7f65e013c220 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65e0138600 con 0x7f65e0135b60 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient(hunting): authenticate will time out at 1173.309952s 2024-09-15T20:31:52.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65ded9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f65e00a24e0 0x7f65e0071aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50954/0 (socket says 172.21.15.78:50954) 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65ded9d640 1 -- 172.21.15.78:0/3043167703 learned_addr learned my addr 172.21.15.78:0/3043167703 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 client.?.objecter ms_handle_connect 0x7f65e00a24e0 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 client.?.objecter ms_handle_connect 0x7f65e0135b60 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1736809273 0 0) 0x7f65e00745e0 con 0x7f65e00a24e0 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): my global_id is 4380 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65c0004ec0 con 0x7f65e00a24e0 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1206057029 0 0) 0x7f65e0138600 con 0x7f65e0135b60 2024-09-15T20:31:52.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): my global_id is 4385 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e00745e0 con 0x7f65e0135b60 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 client.?.objecter ms_handle_connect 0x7f65e013c220 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1926786565 0 0) 0x7f65c0004ec0 con 0x7f65e00a24e0 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65e0138600 con 0x7f65e00a24e0 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65d0001a20 con 0x7f65e00a24e0 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): dump: 2024-09-15T20:31:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3060528612 0 0) 0x7f65e00745e0 con 0x7f65e0135b60 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65c0004ec0 con 0x7f65e0135b60 2024-09-15T20:31:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65c8001260 con 0x7f65e0135b60 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient(hunting): dump: 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.442 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 317032152 0 0) 0x7f65e0138600 con 0x7f65e00a24e0 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 >> v1:172.21.15.78:6790/0 conn(0x7f65e0135b60 legacy=0x7f65e00721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 >> v1:172.21.15.146:6789/0 conn(0x7f65e013c220 legacy=0x7f65e00728c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 monclient: found mon.a 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:52.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65e00a35b0 con 0x7f65e00a24e0 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient: _finish_auth 0 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 5 monclient: authenticate success, global_id 4380 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: _renew_subs 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65e0136e00 con 0x7f65e00a24e0 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 client.4380.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: _renew_subs 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f65e01d3df0 con 0x7f65e00a24e0 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 librados: init done 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65d0001cc0 con 0x7f65e00a24e0 2024-09-15T20:31:52.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65d0003a70 con 0x7f65e00a24e0 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65ddd9b640 4 set_mon_vals no callback set 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 monclient: dump: 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.445 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f65d0003c10 con 0x7f65e00a24e0 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c4005090 con 0x7f65e00a24e0 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(19..19 src has 1..19) ==== 3562+0+0 (unknown 2113909411 0 0) 0x7f65d004b0f0 con 0x7f65e00a24e0 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 10 client.4380.objecter ms_dispatch 0x7f65e0000ba0 osd_map(19..19 src has 1..19) 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 3 client.4380.objecter handle_osd_map got epochs [19,19] > 0 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.423+0000 7f65d7fff640 3 client.4380.objecter handle_osd_map decoding full epoch 19 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.427+0000 7f65d7fff640 20 client.4380.objecter dump_active .. 0 homeless 2024-09-15T20:31:52.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.427+0000 7f65d659c640 10 monclient: get_auth_request con 0x7f65c003e270 auth_method 0 2024-09-15T20:31:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.427+0000 7f65d7fff640 10 client.4380.objecter ms_handle_connect 0x7f65c003e270 2024-09-15T20:31:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.427+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f65e009d180 con 0x7f65e00a24e0 2024-09-15T20:31:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.431+0000 7f65d7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.431+0000 7f65d7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:52.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.591+0000 7f65e51de640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:52.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.591+0000 7f65e51de640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:52.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.591+0000 7f65e51de640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:52.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.591+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65c4005380 con 0x7f65e00a24e0 2024-09-15T20:31:52.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.591+0000 7f65d7fff640 1 -- 172.21.15.78:0/3043167703 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+10894 (unknown 3413328019 0 527973929) 0x7f65d0016b10 con 0x7f65e00a24e0 2024-09-15T20:31:52.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.591+0000 7f65d7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:31:52.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.591+0000 7f65d7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:52.596 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:52.596 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":20,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","created":"2024-09-15T20:31:27.702188+0000","modified":"2024-09-15T20:31:52.456281+0000","last_up_change":"2024-09-15T20:31:37.715594+0000","last_in_change":"2024-09-15T20:31:29.437051+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:31:38.725860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T20:31:44.650592+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9003b16b-6a87-4659-8436-ff541ea8dfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6804","nonce":3648339548}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6805","nonce":3648339548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6807","nonce":3648339548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6806","nonce":3648339548}]},"public_addr":"172.21.15.78:6804/3648339548","cluster_addr":"172.21.15.78:6805/3648339548","heartbeat_back_addr":"172.21.15.78:6807/3648339548","heartbeat_front_addr":"172.21.15.78:6806/3648339548","state":["exists","up"]},{"osd":1,"uuid":"be602c68-218b-4573-8d19-eef568af9d4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6800","nonce":4006351464}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6801","nonce":4006351464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6803","nonce":4006351464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6802","nonce":4006351464}]},"public_addr":"172.21.15.78:6800/4006351464","cluster_addr":"172.21.15.78:6801/4006351464","heartbeat_back_addr":"172.21.15.78:6803/4006351464","heartbeat_front_addr":"172.21.15.78:6802/4006351464","state":["exists","up"]},{"osd":2,"uuid":"1fab92e2-4880-4f66-86f9-1c59752850a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6808","nonce":404661851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6809","nonce":404661851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6811","nonce":404661851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6810","nonce":404661851}]},"public_addr":"172.21.15.78:6808/404661851","cluster_addr":"172.21.15.78:6809/404661851","heartbeat_back_addr":"172.21.15.78:6811/404661851","heartbeat_front_addr":"172.21.15.78:6810/404661851","state":["exists","up"]},{"osd":3,"uuid":"74c2a9ff-0635-4664-9dc5-d156f4b60bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6800","nonce":490126554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6801","nonce":490126554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6803","nonce":490126554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6802","nonce":490126554}]},"public_addr":"172.21.15.146:6800/490126554","cluster_addr":"172.21.15.146:6801/490126554","heartbeat_back_addr":"172.21.15.146:6803/490126554","heartbeat_front_addr":"172.21.15.146:6802/490126554","state":["exists","up"]},{"osd":4,"uuid":"0abc805d-20bc-40e6-a25a-0d9694ace9eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6808","nonce":2960966471}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6809","nonce":2960966471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6811","nonce":2960966471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6810","nonce":2960966471}]},"public_addr":"172.21.15.146:6808/2960966471","cluster_addr":"172.21.15.146:6809/2960966471","heartbeat_back_addr":"172.21.15.146:6811/2960966471","heartbeat_front_addr":"172.21.15.146:6810/2960966471","state":["exists","up"]},{"osd":5,"uuid":"08c34b4f-deb1-401a-aab7-00854ae6712f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6804","nonce":516207226}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6805","nonce":516207226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6807","nonce":516207226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6806","nonce":516207226}]},"public_addr":"172.21.15.146:6804/516207226","cluster_addr":"172.21.15.146:6805/516207226","heartbeat_back_addr":"172.21.15.146:6807/516207226","heartbeat_front_addr":"172.21.15.146:6806/516207226","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.779852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.576609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.081865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.582230+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.088303+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.878442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:31:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 10 librados: watch_flush enter 2024-09-15T20:31:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 10 librados: watch_flush exit 2024-09-15T20:31:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 20 client.4380.objecter shutdown clearing up homeless session... 2024-09-15T20:31:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 10 client.4380.objecter successfully canceled tick 2024-09-15T20:31:52.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 >> v1:172.21.15.146:6812/435598642 conn(0x7f65c003e270 legacy=0x7f65c0040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 10 monclient: shutdown 2024-09-15T20:31:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 >> v1:172.21.15.78:6789/0 conn(0x7f65e00a24e0 legacy=0x7f65e0071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 shutdown_connections 2024-09-15T20:31:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 >> 172.21.15.78:0/3043167703 conn(0x7f65e00a63c0 msgr2=0x7f65e00a4630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 shutdown_connections 2024-09-15T20:31:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 1 -- 172.21.15.78:0/3043167703 wait complete. 2024-09-15T20:31:52.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.595+0000 7f65e51de640 1 librados: shutdown 2024-09-15T20:31:52.613 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-15T20:31:38.725860+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 6.059999942779541, 'score_stable': 6.059999942779541, 'optimal_score': 0.33000001311302185, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2024-09-15T20:31:44.650592+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '19', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 19, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-15T20:31:52.613 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-09-15T20:31:52.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient: get_monmap_and_config 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient: build_initial_monmap 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient: monmap: 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:52.722154+0000 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:52.722154+0000 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:52.723 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 1 Processor -- start 2024-09-15T20:31:52.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 1 -- start start 2024-09-15T20:31:52.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient: init 2024-09-15T20:31:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae40745e0 con 0x7f0ae413c220 2024-09-15T20:31:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae406ecf0 con 0x7f0ae40a24e0 2024-09-15T20:31:52.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae406efd0 con 0x7f0ae4135b60 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0aeb532640 10 monclient(hunting): authenticate will time out at 1173.607581s 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0ae92ce640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f0ae4135b60 0x7f0ae4135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54788/0 (socket says 172.21.15.78:54788) 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0ae92ce640 1 -- 172.21.15.78:0/1277397083 learned_addr learned my addr 172.21.15.78:0/1277397083 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1274448167 0 0) 0x7f0ae40745e0 con 0x7f0ae413c220 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 10 monclient(hunting): my global_id is 4383 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ad0003530 con 0x7f0ae413c220 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4280228044 0 0) 0x7f0ae406efd0 con 0x7f0ae4135b60 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 10 monclient(hunting): my global_id is 4388 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.719+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ae40745e0 con 0x7f0ae4135b60 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3366949177 0 0) 0x7f0ad0003530 con 0x7f0ae413c220 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 >> v1:172.21.15.78:6790/0 conn(0x7f0ae4135b60 legacy=0x7f0ae4135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 >> v1:172.21.15.146:6789/0 conn(0x7f0ae40a24e0 legacy=0x7f0ae413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 1 monclient: found mon.noname-a 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae4136650 con 0x7f0ae413c220 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: _finish_auth 0 2024-09-15T20:31:52.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0adc001a20 con 0x7f0ae413c220 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 5 monclient: authenticate success, global_id 4383 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: dump: 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.730 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0adc001c40 con 0x7f0ae413c220 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 1 -- 172.21.15.78:0/1277397083 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0adc002010 con 0x7f0ae413c220 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adbfff640 10 monclient: dump: 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.731 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 4 set_mon_vals no callback set 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: shutdown 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- 172.21.15.78:0/1277397083 >> v1:172.21.15.78:6789/0 conn(0x7f0ae413c220 legacy=0x7f0ae413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- 172.21.15.78:0/1277397083 shutdown_connections 2024-09-15T20:31:52.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- 172.21.15.78:0/1277397083 >> 172.21.15.78:0/1277397083 conn(0x7f0ae40a63c0 msgr2=0x7f0ae40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- 172.21.15.78:0/1277397083 shutdown_connections 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- 172.21.15.78:0/1277397083 wait complete. 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: build_initial_monmap 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: monmap: 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:52.733 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 librados: starting msgr at 2024-09-15T20:31:52.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 librados: starting objecter 2024-09-15T20:31:52.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 Processor -- start 2024-09-15T20:31:52.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- start start 2024-09-15T20:31:52.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 librados: setting wanted keys 2024-09-15T20:31:52.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 librados: calling monclient init 2024-09-15T20:31:52.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: init 2024-09-15T20:31:52.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient(hunting): picked mon.noname-a con 0x7f0ae40a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient(hunting): picked mon.noname-b con 0x7f0ae4135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient(hunting): picked mon.noname-c con 0x7f0ae413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae4071640 con 0x7f0ae40a24e0 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae409e2b0 con 0x7f0ae4135b60 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ae4071920 con 0x7f0ae413c220 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient(hunting): authenticate will time out at 1173.610425s 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ae92ce640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0ae40a24e0 0x7f0ae409d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50966/0 (socket says 172.21.15.78:50966) 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ae92ce640 1 -- 172.21.15.78:0/555989473 learned_addr learned my addr 172.21.15.78:0/555989473 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 client.?.objecter ms_handle_connect 0x7f0ae413c220 2024-09-15T20:31:52.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 client.?.objecter ms_handle_connect 0x7f0ae40a24e0 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3402485693 0 0) 0x7f0ae4071640 con 0x7f0ae40a24e0 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): my global_id is 4386 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ac8004ef0 con 0x7f0ae40a24e0 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 66501828 0 0) 0x7f0ae4071920 con 0x7f0ae413c220 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): my global_id is 4391 2024-09-15T20:31:52.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ae4071640 con 0x7f0ae413c220 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 client.?.objecter ms_handle_connect 0x7f0ae4135b60 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3587962526 0 0) 0x7f0ae4071640 con 0x7f0ae413c220 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ae4071920 con 0x7f0ae413c220 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0adc001df0 con 0x7f0ae413c220 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): dump: 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.740 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3015142333 0 0) 0x7f0ac8004ef0 con 0x7f0ae40a24e0 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0ae4071640 con 0x7f0ae40a24e0 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0ad4001940 con 0x7f0ae40a24e0 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): dump: 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.741 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 837841203 0 0) 0x7f0ae409e2b0 con 0x7f0ae4135b60 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient(hunting): my global_id is 4390 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ac8004ef0 con 0x7f0ae4135b60 2024-09-15T20:31:52.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3230604774 0 0) 0x7f0ae4071920 con 0x7f0ae413c220 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 >> v1:172.21.15.146:6789/0 conn(0x7f0ae4135b60 legacy=0x7f0ae409dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 >> v1:172.21.15.78:6789/0 conn(0x7f0ae40a24e0 legacy=0x7f0ae409d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 monclient: found mon.c 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ae413bdc0 con 0x7f0ae413c220 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 5 monclient: authenticate success, global_id 4391 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: _renew_subs 2024-09-15T20:31:52.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0ae409d180 con 0x7f0ae413c220 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 client.4391.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: _renew_subs 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0ae413af70 con 0x7f0ae413c220 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0aeb532640 1 librados: init done 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0adc0020b0 con 0x7f0ae413c220 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0adc003cf0 con 0x7f0ae413c220 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0ad9ffb640 10 monclient: dump: 2024-09-15T20:31:52.744 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.723+0000 7f0adb7fe640 4 set_mon_vals no callback set 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0aeb532640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0aeb532640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0aeb532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ac4005090 con 0x7f0ae413c220 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f0adc00fd60 con 0x7f0ae413c220 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7f0adc04a0e0 con 0x7f0ae413c220 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0ad9ffb640 10 client.4391.objecter ms_dispatch 0x7f0ae4000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0ad9ffb640 3 client.4391.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0ad9ffb640 3 client.4391.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0ad9ffb640 20 client.4391.objecter dump_active .. 0 homeless 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0ae92ce640 10 monclient: get_auth_request con 0x7f0ac803e1d0 auth_method 0 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.727+0000 7f0ad9ffb640 10 client.4391.objecter ms_handle_connect 0x7f0ac803e1d0 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.731+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f0adc015bc0 con 0x7f0ae413c220 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.731+0000 7f0ad9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:52.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.731+0000 7f0ad9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.887+0000 7f0aeb532640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T20:31:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.887+0000 7f0aeb532640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T20:31:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.887+0000 7f0aeb532640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:52.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.887+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f0ac4005c50 con 0x7f0ae413c220 2024-09-15T20:31:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0ad9ffb640 1 -- 172.21.15.78:0/555989473 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v20) ==== 93+0+10 (unknown 3471955896 0 2170607528) 0x7f0adc0154c0 con 0x7f0ae413c220 2024-09-15T20:31:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0ad9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}] 2024-09-15T20:31:52.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0ad9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:52.894 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 1 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 10 librados: watch_flush enter 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 10 librados: watch_flush exit 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 20 client.4391.objecter shutdown clearing up homeless session... 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 10 client.4391.objecter successfully canceled tick 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 >> v1:172.21.15.146:6812/435598642 conn(0x7f0ac803e1d0 legacy=0x7f0ac8040690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 10 monclient: shutdown 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 >> v1:172.21.15.78:6790/0 conn(0x7f0ae413c220 legacy=0x7f0ae41d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:52.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 shutdown_connections 2024-09-15T20:31:52.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 >> 172.21.15.78:0/555989473 conn(0x7f0ae40a63c0 msgr2=0x7f0ae412b7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:52.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 shutdown_connections 2024-09-15T20:31:52.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 1 -- 172.21.15.78:0/555989473 wait complete. 2024-09-15T20:31:52.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:52.891+0000 7f0aeb532640 1 librados: shutdown 2024-09-15T20:31:52.910 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient: get_monmap_and_config 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient: build_initial_monmap 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient: monmap: 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:53.030322+0000 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:53.030322+0000 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:53.032 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 1 Processor -- start 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 1 -- start start 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient: init 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient(hunting): picked mon.noname-b con 0x7f8b0c134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient(hunting): picked mon.noname-c con 0x7f8b0c09f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient(hunting): picked mon.noname-a con 0x7f8b0c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b0c0a0470 con 0x7f8b0c06e960 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b0c09feb0 con 0x7f8b0c09f5a0 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b0c09e4a0 con 0x7f8b0c134bb0 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient(hunting): authenticate will time out at 1173.914953s 2024-09-15T20:31:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b11f8b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f8b0c134bb0 0x7f8b0c134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54808/0 (socket says 172.21.15.78:54808) 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b11f8b640 1 -- 172.21.15.78:0/2974801612 learned_addr learned my addr 172.21.15.78:0/2974801612 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 875734586 0 0) 0x7f8b0c09e4a0 con 0x7f8b0c134bb0 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient(hunting): my global_id is 4394 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8af8003530 con 0x7f8b0c134bb0 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2454945720 0 0) 0x7f8b040014a0 con 0x7f8b0c06e960 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient(hunting): my global_id is 4389 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b0c09e4a0 con 0x7f8b0c06e960 2024-09-15T20:31:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1516962050 0 0) 0x7f8af8003530 con 0x7f8b0c134bb0 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 >> v1:172.21.15.146:6789/0 conn(0x7f8b0c09f5a0 legacy=0x7f8b0c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 >> v1:172.21.15.78:6789/0 conn(0x7f8b0c06e960 legacy=0x7f8b0c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: _finish_hunting 0 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 monclient: found mon.noname-b 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0c0a0ff0 con 0x7f8b0c134bb0 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: _finish_auth 0 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: _check_auth_tickets 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8b080016a0 con 0x7f8b0c134bb0 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 5 monclient: authenticate success, global_id 4394 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: dump: 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b08001840 con 0x7f8b0c134bb0 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:53.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 1 -- 172.21.15.78:0/2974801612 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8b08001cd0 con 0x7f8b0c134bb0 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b141ef640 4 set_mon_vals no callback set 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.027+0000 7f8b10f89640 10 monclient: dump: 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: shutdown 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- 172.21.15.78:0/2974801612 >> v1:172.21.15.78:6790/0 conn(0x7f8b0c134bb0 legacy=0x7f8b0c134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- 172.21.15.78:0/2974801612 shutdown_connections 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- 172.21.15.78:0/2974801612 >> 172.21.15.78:0/2974801612 conn(0x7f8b0c0a63c0 msgr2=0x7f8b0c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- 172.21.15.78:0/2974801612 shutdown_connections 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- 172.21.15.78:0/2974801612 wait complete. 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: build_initial_monmap 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: monmap: 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:53.040 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 librados: starting msgr at 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 librados: starting objecter 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 Processor -- start 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- start start 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 librados: setting wanted keys 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 librados: calling monclient init 2024-09-15T20:31:53.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: init 2024-09-15T20:31:53.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient(hunting): picked mon.noname-a con 0x7f8b0c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient(hunting): picked mon.noname-b con 0x7f8b0c09f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient(hunting): picked mon.noname-c con 0x7f8b0c134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b0c073c40 con 0x7f8b0c06e960 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b0c128240 con 0x7f8b0c09f5a0 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8b0c09d180 con 0x7f8b0c134bb0 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient(hunting): authenticate will time out at 1173.917929s 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b11f8b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8b0c06e960 0x7f8b0c133af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50972/0 (socket says 172.21.15.78:50972) 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b11f8b640 1 -- 172.21.15.78:0/407819352 learned_addr learned my addr 172.21.15.78:0/407819352 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 client.?.objecter ms_handle_connect 0x7f8b0c06e960 2024-09-15T20:31:53.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 client.?.objecter ms_handle_connect 0x7f8b0c134bb0 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1048042029 0 0) 0x7f8b0c0a0ff0 con 0x7f8b0c06e960 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): my global_id is 4392 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8af0004ec0 con 0x7f8b0c06e960 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1043654178 0 0) 0x7f8b0c09d180 con 0x7f8b0c134bb0 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): my global_id is 4397 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8b0c0a0ff0 con 0x7f8b0c134bb0 2024-09-15T20:31:53.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4105129633 0 0) 0x7f8af0004ec0 con 0x7f8b0c06e960 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8b0c09d180 con 0x7f8b0c06e960 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8b08001b90 con 0x7f8b0c06e960 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): dump: 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.047 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1353779618 0 0) 0x7f8b0c0a0ff0 con 0x7f8b0c134bb0 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8af0004ec0 con 0x7f8b0c134bb0 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8b04001910 con 0x7f8b0c134bb0 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient(hunting): dump: 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.048 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 553179012 0 0) 0x7f8b0c09d180 con 0x7f8b0c06e960 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 >> v1:172.21.15.78:6790/0 conn(0x7f8b0c134bb0 legacy=0x7f8b0c1d52d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 >> v1:172.21.15.146:6789/0 conn(0x7f8b0c09f5a0 legacy=0x7f8b0c1d1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 monclient: found mon.a 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b0c127f60 con 0x7f8b0c06e960 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 5 monclient: authenticate success, global_id 4392 2024-09-15T20:31:53.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: _renew_subs 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8b0c126d20 con 0x7f8b0c06e960 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 client.4392.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: _renew_subs 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8b0c1cccb0 con 0x7f8b0c06e960 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 librados: init done 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8b080017a0 con 0x7f8b0c06e960 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8b08004da0 con 0x7f8b0c06e960 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b03fff640 4 set_mon_vals no callback set 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 monclient: dump: 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.050 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f8b08010e10 con 0x7f8b0c06e960 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8af4005090 con 0x7f8b0c06e960 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7f8b0804b010 con 0x7f8b0c06e960 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 10 client.4392.objecter ms_dispatch 0x7f8b0c000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 3 client.4392.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 3 client.4392.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.031+0000 7f8b027fc640 20 client.4392.objecter dump_active .. 0 homeless 2024-09-15T20:31:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.035+0000 7f8b1178a640 10 monclient: get_auth_request con 0x7f8af003e030 auth_method 0 2024-09-15T20:31:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.035+0000 7f8b027fc640 10 client.4392.objecter ms_handle_connect 0x7f8af003e030 2024-09-15T20:31:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.039+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f8b0c073c40 con 0x7f8b0c06e960 2024-09-15T20:31:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.039+0000 7f8b027fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.039+0000 7f8b027fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:53.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.207+0000 7f8b141ef640 10 start_mon_command cmd=[{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T20:31:53.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.207+0000 7f8b141ef640 10 monclient: _send_command 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T20:31:53.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.207+0000 7f8b141ef640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:53.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.207+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) -- 0x7f8af4005c50 con 0x7f8b0c06e960 2024-09-15T20:31:53.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b027fc640 1 -- 172.21.15.78:0/407819352 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v20) ==== 92+0+10 (unknown 3412311620 0 265718887) 0x7f8b08016a90 con 0x7f8b0c06e960 2024-09-15T20:31:53.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b027fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}] 2024-09-15T20:31:53.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b027fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:53.214 INFO:teuthology.orchestra.run.smithi078.stdout:pg_num: 8 2024-09-15T20:31:53.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 10 librados: watch_flush enter 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 10 librados: watch_flush exit 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 20 client.4392.objecter shutdown clearing up homeless session... 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 10 client.4392.objecter successfully canceled tick 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 >> v1:172.21.15.146:6812/435598642 conn(0x7f8af003e030 legacy=0x7f8af00404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 10 monclient: shutdown 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 >> v1:172.21.15.78:6789/0 conn(0x7f8b0c06e960 legacy=0x7f8b0c133af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 shutdown_connections 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 >> 172.21.15.78:0/407819352 conn(0x7f8b0c0a63c0 msgr2=0x7f8b0c09d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 shutdown_connections 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 1 -- 172.21.15.78:0/407819352 wait complete. 2024-09-15T20:31:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.211+0000 7f8b141ef640 1 librados: shutdown 2024-09-15T20:31:53.233 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-09-15T20:31:53.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.343+0000 7fd9d0569640 10 monclient: get_monmap_and_config 2024-09-15T20:31:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.343+0000 7fd9d0569640 10 monclient: build_initial_monmap 2024-09-15T20:31:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.343+0000 7fd9d0569640 10 monclient: monmap: 2024-09-15T20:31:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.348 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:53.347731+0000 2024-09-15T20:31:53.349 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:53.347731+0000 2024-09-15T20:31:53.349 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:53.349 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.349 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:53.349 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:53.349 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:53.349 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.343+0000 7fd9d0569640 1 Processor -- start 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.343+0000 7fd9d0569640 1 -- start start 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.343+0000 7fd9d0569640 10 monclient: init 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): picked mon.noname-b con 0x7fd9c8137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): picked mon.noname-a con 0x7fd9c8074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): picked mon.noname-c con 0x7fd9c813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c8138850 con 0x7fd9c8074500 2024-09-15T20:31:53.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c80a5da0 con 0x7fd9c813e370 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c8073560 con 0x7fd9c8137d60 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): authenticate will time out at 1174.232391s 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cdb04640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd9c8074500 0x7fd9c813cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50988/0 (socket says 172.21.15.78:50988) 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cdb04640 1 -- 172.21.15.78:0/1067171093 learned_addr learned my addr 172.21.15.78:0/1067171093 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9ce305640 1 --1- 172.21.15.78:0/1067171093 >> v1:172.21.15.78:6790/0 conn(0x7fd9c8137d60 0x7fd9c8138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54818/0 (socket says 172.21.15.78:54818) 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3030063800 0 0) 0x7fd9c8138850 con 0x7fd9c8074500 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient(hunting): my global_id is 4395 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9b4003530 con 0x7fd9c8074500 2024-09-15T20:31:53.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 657932450 0 0) 0x7fd9c8073560 con 0x7fd9c8137d60 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient(hunting): my global_id is 4400 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c8138850 con 0x7fd9c8137d60 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2825275023 0 0) 0x7fd9b4003530 con 0x7fd9c8074500 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 >> v1:172.21.15.78:6790/0 conn(0x7fd9c8137d60 legacy=0x7fd9c8138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 >> v1:172.21.15.146:6789/0 conn(0x7fd9c813e370 legacy=0x7fd9c8140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: _finish_hunting 0 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 monclient: found mon.noname-a 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:53.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c8073840 con 0x7fd9c8074500 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: _finish_auth 0 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: _check_auth_tickets 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd9bc0016a0 con 0x7fd9c8074500 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 5 monclient: authenticate success, global_id 4395 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: dump: 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.359 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9bc001840 con 0x7fd9c8074500 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 4 set_mon_vals no callback set 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 1 -- 172.21.15.78:0/1067171093 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd9bc001c10 con 0x7fd9c8074500 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:53.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9cd303640 10 monclient: dump: 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: shutdown 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- 172.21.15.78:0/1067171093 >> v1:172.21.15.78:6789/0 conn(0x7fd9c8074500 legacy=0x7fd9c813cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- 172.21.15.78:0/1067171093 shutdown_connections 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- 172.21.15.78:0/1067171093 >> 172.21.15.78:0/1067171093 conn(0x7fd9c80aa650 msgr2=0x7fd9c80aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- 172.21.15.78:0/1067171093 shutdown_connections 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- 172.21.15.78:0/1067171093 wait complete. 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: build_initial_monmap 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: monmap: 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:53.362 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 librados: starting msgr at 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 librados: starting objecter 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 Processor -- start 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- start start 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 librados: setting wanted keys 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 librados: calling monclient init 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: init 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): picked mon.noname-c con 0x7fd9c8074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:53.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): picked mon.noname-a con 0x7fd9c8137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): picked mon.noname-b con 0x7fd9c813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c8139840 con 0x7fd9c8137d60 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c8139c90 con 0x7fd9c813e370 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9c813b750 con 0x7fd9c8074500 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9d0569640 10 monclient(hunting): authenticate will time out at 1174.235397s 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9ce305640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd9c8074500 0x7fd9c8138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54832/0 (socket says 172.21.15.78:54832) 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9ce305640 1 -- 172.21.15.78:0/3882404447 learned_addr learned my addr 172.21.15.78:0/3882404447 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 client.?.objecter ms_handle_connect 0x7fd9c8074500 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 client.?.objecter ms_handle_connect 0x7fd9c8137d60 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2242588021 0 0) 0x7fd9c813b750 con 0x7fd9c8074500 2024-09-15T20:31:53.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 monclient(hunting): my global_id is 4403 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9ac004ec0 con 0x7fd9c8074500 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3058887074 0 0) 0x7fd9c8139840 con 0x7fd9c8137d60 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 monclient(hunting): my global_id is 4398 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9c813b750 con 0x7fd9c8137d60 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 client.?.objecter ms_handle_connect 0x7fd9c813e370 2024-09-15T20:31:53.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.347+0000 7fd9baffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4278041230 0 0) 0x7fd9c813b750 con 0x7fd9c8137d60 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9c8139840 con 0x7fd9c8137d60 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 897297459 0 0) 0x7fd9ac004ec0 con 0x7fd9c8074500 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9c813b750 con 0x7fd9c8074500 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd9bc001560 con 0x7fd9c8137d60 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): dump: 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.366 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd9c4001d00 con 0x7fd9c8074500 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): dump: 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.367 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3264898406 0 0) 0x7fd9c8139c90 con 0x7fd9c813e370 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient(hunting): my global_id is 4399 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9ac004ec0 con 0x7fd9c813e370 2024-09-15T20:31:53.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3801179113 0 0) 0x7fd9c813b750 con 0x7fd9c8074500 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 >> v1:172.21.15.146:6789/0 conn(0x7fd9c813e370 legacy=0x7fd9c81d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 >> v1:172.21.15.78:6789/0 conn(0x7fd9c8137d60 legacy=0x7fd9c8139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 monclient: found mon.c 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9c813ba80 con 0x7fd9c8074500 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient: _finish_auth 0 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 5 monclient: authenticate success, global_id 4403 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 10 monclient: _renew_subs 2024-09-15T20:31:53.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd9c813ab70 con 0x7fd9c8074500 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 10 client.4403.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 10 monclient: _renew_subs 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd9c81412e0 con 0x7fd9c8074500 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 1 librados: init done 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9c4002020 con 0x7fd9c8074500 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd9c4004fc0 con 0x7fd9c8074500 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9ccb02640 4 set_mon_vals no callback set 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 monclient: dump: 2024-09-15T20:31:53.370 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9b0005090 con 0x7fd9c8074500 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd9c4011030 con 0x7fd9c8074500 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7fd9c404b400 con 0x7fd9c8074500 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 client.4403.objecter ms_dispatch 0x7fd9c8000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 3 client.4403.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 3 client.4403.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 20 client.4403.objecter dump_active .. 0 homeless 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9cdb04640 10 monclient: get_auth_request con 0x7fd9ac03e1d0 auth_method 0 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.351+0000 7fd9baffd640 10 client.4403.objecter ms_handle_connect 0x7fd9ac03e1d0 2024-09-15T20:31:53.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.355+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd9c4016e80 con 0x7fd9c8074500 2024-09-15T20:31:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.355+0000 7fd9baffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.355+0000 7fd9baffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:53.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.523+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd9b0000ba0 con 0x7fd9ac03e1d0 2024-09-15T20:31:53.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9baffd640 1 -- 172.21.15.78:0/3882404447 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34191 (unknown 2965378022 0 1806742744) 0x7fd9b0000ba0 con 0x7fd9ac03e1d0 2024-09-15T20:31:53.530 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:53.531 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 10 librados: watch_flush enter 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 10 librados: watch_flush exit 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 20 client.4403.objecter shutdown clearing up homeless session... 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 10 client.4403.objecter successfully canceled tick 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 >> v1:172.21.15.146:6812/435598642 conn(0x7fd9ac03e1d0 legacy=0x7fd9ac040690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 10 monclient: shutdown 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 >> v1:172.21.15.78:6790/0 conn(0x7fd9c8074500 legacy=0x7fd9c8138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 shutdown_connections 2024-09-15T20:31:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 >> 172.21.15.78:0/3882404447 conn(0x7fd9c80aa650 msgr2=0x7fd9c80a3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 shutdown_connections 2024-09-15T20:31:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 1 -- 172.21.15.78:0/3882404447 wait complete. 2024-09-15T20:31:53.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.527+0000 7fd9d0569640 1 librados: shutdown 2024-09-15T20:31:53.548 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-15T20:31:52.654898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163272,"kb_used_data":1088,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067768,"statfs":{"total":579820584960,"available":579653394432,"internally_reserved":0,"allocated":1114112,"data_stored":1371166,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001669"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825675+0000","last_change":"2024-09-15T20:31:47.826026+0000","last_active":"2024-09-15T20:31:47.825675+0000","last_peered":"2024-09-15T20:31:47.825675+0000","last_clean":"2024-09-15T20:31:47.825675+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:31:47.825675+0000","last_undegraded":"2024-09-15T20:31:47.825675+0000","last_fullsized":"2024-09-15T20:31:47.825675+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:19.315897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837983+0000","last_change":"2024-09-15T20:31:47.838456+0000","last_active":"2024-09-15T20:31:47.837983+0000","last_peered":"2024-09-15T20:31:47.837983+0000","last_clean":"2024-09-15T20:31:47.837983+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:31:47.837983+0000","last_undegraded":"2024-09-15T20:31:47.837983+0000","last_fullsized":"2024-09-15T20:31:47.837983+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:31:54.370290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825214+0000","last_change":"2024-09-15T20:31:47.825665+0000","last_active":"2024-09-15T20:31:47.825214+0000","last_peered":"2024-09-15T20:31:47.825214+0000","last_clean":"2024-09-15T20:31:47.825214+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:31:47.825214+0000","last_undegraded":"2024-09-15T20:31:47.825214+0000","last_fullsized":"2024-09-15T20:31:47.825214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:40:22.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837836+0000","last_change":"2024-09-15T20:31:47.838237+0000","last_active":"2024-09-15T20:31:47.837836+0000","last_peered":"2024-09-15T20:31:47.837836+0000","last_clean":"2024-09-15T20:31:47.837836+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:31:47.837836+0000","last_undegraded":"2024-09-15T20:31:47.837836+0000","last_fullsized":"2024-09-15T20:31:47.837836+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:49.045779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837218+0000","last_change":"2024-09-15T20:31:47.834864+0000","last_active":"2024-09-15T20:31:47.837218+0000","last_peered":"2024-09-15T20:31:47.837218+0000","last_clean":"2024-09-15T20:31:47.837218+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:31:47.837218+0000","last_undegraded":"2024-09-15T20:31:47.837218+0000","last_fullsized":"2024-09-15T20:31:47.837218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:56.096771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.969706+0000","last_change":"2024-09-15T20:31:47.969706+0000","last_active":"2024-09-15T20:31:47.969706+0000","last_peered":"2024-09-15T20:31:47.969706+0000","last_clean":"2024-09-15T20:31:47.969706+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:31:47.969706+0000","last_undegraded":"2024-09-15T20:31:47.969706+0000","last_fullsized":"2024-09-15T20:31:47.969706+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:48.645009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825141+0000","last_change":"2024-09-15T20:31:47.825565+0000","last_active":"2024-09-15T20:31:47.825141+0000","last_peered":"2024-09-15T20:31:47.825141+0000","last_clean":"2024-09-15T20:31:47.825141+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:31:47.825141+0000","last_undegraded":"2024-09-15T20:31:47.825141+0000","last_fullsized":"2024-09-15T20:31:47.825141+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:49:12.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.833645+0000","last_change":"2024-09-15T20:31:47.833645+0000","last_active":"2024-09-15T20:31:47.833645+0000","last_peered":"2024-09-15T20:31:47.833645+0000","last_clean":"2024-09-15T20:31:47.833645+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:31:47.833645+0000","last_undegraded":"2024-09-15T20:31:47.833645+0000","last_fullsized":"2024-09-15T20:31:47.833645+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:24.343248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.824491+0000","last_change":"2024-09-15T20:31:40.658325+0000","last_active":"2024-09-15T20:31:47.824491+0000","last_peered":"2024-09-15T20:31:47.824491+0000","last_clean":"2024-09-15T20:31:47.824491+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:31:47.824491+0000","last_undegraded":"2024-09-15T20:31:47.824491+0000","last_fullsized":"2024-09-15T20:31:47.824491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_clean_scrub_stamp":"2024-09-15T20:31:38.726961+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:41.743437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":52212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":581140,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":581140,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:53.550 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:31:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.663+0000 7f5ae9400640 10 monclient: get_monmap_and_config 2024-09-15T20:31:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.663+0000 7f5ae9400640 10 monclient: build_initial_monmap 2024-09-15T20:31:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.663+0000 7f5ae9400640 10 monclient: monmap: 2024-09-15T20:31:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:53.667679+0000 2024-09-15T20:31:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:53.667679+0000 2024-09-15T20:31:53.669 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:53.669 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.669 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:53.669 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:53.669 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:53.669 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.663+0000 7f5ae9400640 1 Processor -- start 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.663+0000 7f5ae9400640 1 -- start start 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.663+0000 7f5ae9400640 10 monclient: init 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae4137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae4074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae413e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae4138850 con 0x7f5ae4074500 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae40a5da0 con 0x7f5ae4137d60 2024-09-15T20:31:53.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae4073560 con 0x7f5ae413e370 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): authenticate will time out at 1174.552357s 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae37fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f5ae413e370 0x7f5ae4140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54834/0 (socket says 172.21.15.78:54834) 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae37fe640 1 -- 172.21.15.78:0/3585887265 learned_addr learned my addr 172.21.15.78:0/3585887265 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3501250467 0 0) 0x7f5ae4073560 con 0x7f5ae413e370 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient(hunting): my global_id is 4406 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ad0003530 con 0x7f5ae413e370 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2584654922 0 0) 0x7f5ae4138850 con 0x7f5ae4074500 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient(hunting): my global_id is 4401 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae4073560 con 0x7f5ae4074500 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2653287419 0 0) 0x7f5ad0003530 con 0x7f5ae413e370 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 >> v1:172.21.15.146:6789/0 conn(0x7f5ae4137d60 legacy=0x7f5ae4138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 >> v1:172.21.15.78:6789/0 conn(0x7f5ae4074500 legacy=0x7f5ae413cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 monclient: found mon.noname-b 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae4073840 con 0x7f5ae413e370 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:53.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5adc001a20 con 0x7f5ae413e370 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 5 monclient: authenticate success, global_id 4406 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: dump: 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.679 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5adc001bc0 con 0x7f5ae413e370 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 1 -- 172.21.15.78:0/3585887265 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5adc001fd0 con 0x7f5ae413e370 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 4 set_mon_vals no callback set 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae1ffb640 10 monclient: dump: 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.680 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: shutdown 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- 172.21.15.78:0/3585887265 >> v1:172.21.15.78:6790/0 conn(0x7f5ae413e370 legacy=0x7f5ae4140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- 172.21.15.78:0/3585887265 shutdown_connections 2024-09-15T20:31:53.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- 172.21.15.78:0/3585887265 >> 172.21.15.78:0/3585887265 conn(0x7f5ae40aa650 msgr2=0x7f5ae40aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- 172.21.15.78:0/3585887265 shutdown_connections 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- 172.21.15.78:0/3585887265 wait complete. 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: build_initial_monmap 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: monmap: 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:53.682 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 librados: starting msgr at 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 librados: starting objecter 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 Processor -- start 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- start start 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 librados: setting wanted keys 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 librados: calling monclient init 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: init 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): picked mon.noname-c con 0x7f5ae4074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): picked mon.noname-b con 0x7f5ae4137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): picked mon.noname-a con 0x7f5ae413e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:53.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae413c630 con 0x7f5ae413e370 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae413a2d0 con 0x7f5ae4137d60 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ae413bd90 con 0x7f5ae4074500 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae9400640 10 monclient(hunting): authenticate will time out at 1174.555365s 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5ae413e370 0x7f5ae4139630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51028/0 (socket says 172.21.15.78:51028) 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5ae37fe640 1 -- 172.21.15.78:0/387725863 learned_addr learned my addr 172.21.15.78:0/387725863 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 client.?.objecter ms_handle_connect 0x7f5ae4074500 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 client.?.objecter ms_handle_connect 0x7f5ae413e370 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 820523322 0 0) 0x7f5ae413bd90 con 0x7f5ae4074500 2024-09-15T20:31:53.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): my global_id is 4409 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ac4004ec0 con 0x7f5ae4074500 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1234784107 0 0) 0x7f5ae413c630 con 0x7f5ae413e370 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): my global_id is 4404 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae413bd90 con 0x7f5ae413e370 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 client.?.objecter ms_handle_connect 0x7f5ae4137d60 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:53.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3486982611 0 0) 0x7f5ac4004ec0 con 0x7f5ae4074500 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ae413c630 con 0x7f5ae4074500 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5acc001940 con 0x7f5ae4074500 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): dump: 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.686 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.667+0000 7f5adb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 351172248 0 0) 0x7f5ae413bd90 con 0x7f5ae413e370 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5ac4004ec0 con 0x7f5ae413e370 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5adc001f10 con 0x7f5ae413e370 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient(hunting): dump: 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.687 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1707837560 0 0) 0x7f5ae413a2d0 con 0x7f5ae4137d60 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient(hunting): my global_id is 4405 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ae413bd90 con 0x7f5ae4137d60 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2964798550 0 0) 0x7f5ae413c630 con 0x7f5ae4074500 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 >> v1:172.21.15.146:6789/0 conn(0x7f5ae4137d60 legacy=0x7f5ae4138f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 >> v1:172.21.15.78:6789/0 conn(0x7f5ae413e370 legacy=0x7f5ae4139630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 monclient: found mon.c 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ae413c070 con 0x7f5ae4074500 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 5 monclient: authenticate success, global_id 4409 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 10 monclient: _renew_subs 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ae413ae30 con 0x7f5ae4074500 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 10 client.4409.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 10 monclient: _renew_subs 2024-09-15T20:31:53.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ae41d5fe0 con 0x7f5ae4074500 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 1 librados: init done 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5acc003600 con 0x7f5ae4074500 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5acc003950 con 0x7f5ae4074500 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 monclient: dump: 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:53.690 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae17fa640 4 set_mon_vals no callback set 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f5acc00f9c0 con 0x7f5ae4074500 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7f5acc049e10 con 0x7f5ae4074500 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 client.4409.objecter ms_dispatch 0x7f5ae4000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 3 client.4409.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 3 client.4409.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ae40008e0 con 0x7f5ae4074500 2024-09-15T20:31:53.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 20 client.4409.objecter dump_active .. 0 homeless 2024-09-15T20:31:53.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5ae27fc640 10 monclient: get_auth_request con 0x7f5ac403e230 auth_method 0 2024-09-15T20:31:53.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.671+0000 7f5adb7fe640 10 client.4409.objecter ms_handle_connect 0x7f5ac403e230 2024-09-15T20:31:53.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.675+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5acc015920 con 0x7f5ae4074500 2024-09-15T20:31:53.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.675+0000 7f5adb7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:53.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.675+0000 7f5adb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:53.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.839+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5ae413b210 con 0x7f5ac403e230 2024-09-15T20:31:53.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.843+0000 7f5adb7fe640 1 -- 172.21.15.78:0/387725863 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+34191 (unknown 2965378022 0 1806742744) 0x7f5ad40010f0 con 0x7f5ac403e230 2024-09-15T20:31:53.847 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:31:53.848 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.843+0000 7f5ae9400640 10 librados: watch_flush enter 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.843+0000 7f5ae9400640 10 librados: watch_flush exit 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.843+0000 7f5ae9400640 20 client.4409.objecter shutdown clearing up homeless session... 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.843+0000 7f5ae9400640 10 client.4409.objecter successfully canceled tick 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 >> v1:172.21.15.146:6812/435598642 conn(0x7f5ac403e230 legacy=0x7f5ac40406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 10 monclient: shutdown 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 >> v1:172.21.15.78:6790/0 conn(0x7f5ae4074500 legacy=0x7f5ae4138810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 shutdown_connections 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 >> 172.21.15.78:0/387725863 conn(0x7f5ae40aa650 msgr2=0x7f5ae40a2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:53.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 shutdown_connections 2024-09-15T20:31:53.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 1 -- 172.21.15.78:0/387725863 wait complete. 2024-09-15T20:31:53.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.847+0000 7f5ae9400640 1 librados: shutdown 2024-09-15T20:31:53.865 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-09-15T20:31:52.654898+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":103,"num_write_kb":1630,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":163272,"kb_used_data":1088,"kb_used_omap":9,"kb_used_meta":162038,"kb_avail":566067768,"statfs":{"total":579820584960,"available":579653394432,"internally_reserved":0,"allocated":1114112,"data_stored":1371166,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":165927584},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001669"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825675+0000","last_change":"2024-09-15T20:31:47.826026+0000","last_active":"2024-09-15T20:31:47.825675+0000","last_peered":"2024-09-15T20:31:47.825675+0000","last_clean":"2024-09-15T20:31:47.825675+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:31:47.825675+0000","last_undegraded":"2024-09-15T20:31:47.825675+0000","last_fullsized":"2024-09-15T20:31:47.825675+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:19.315897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837983+0000","last_change":"2024-09-15T20:31:47.838456+0000","last_active":"2024-09-15T20:31:47.837983+0000","last_peered":"2024-09-15T20:31:47.837983+0000","last_clean":"2024-09-15T20:31:47.837983+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:31:47.837983+0000","last_undegraded":"2024-09-15T20:31:47.837983+0000","last_fullsized":"2024-09-15T20:31:47.837983+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:31:54.370290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825214+0000","last_change":"2024-09-15T20:31:47.825665+0000","last_active":"2024-09-15T20:31:47.825214+0000","last_peered":"2024-09-15T20:31:47.825214+0000","last_clean":"2024-09-15T20:31:47.825214+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:31:47.825214+0000","last_undegraded":"2024-09-15T20:31:47.825214+0000","last_fullsized":"2024-09-15T20:31:47.825214+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:40:22.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837836+0000","last_change":"2024-09-15T20:31:47.838237+0000","last_active":"2024-09-15T20:31:47.837836+0000","last_peered":"2024-09-15T20:31:47.837836+0000","last_clean":"2024-09-15T20:31:47.837836+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:31:47.837836+0000","last_undegraded":"2024-09-15T20:31:47.837836+0000","last_fullsized":"2024-09-15T20:31:47.837836+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:49.045779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"19'2","reported_seq":22,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.837218+0000","last_change":"2024-09-15T20:31:47.834864+0000","last_active":"2024-09-15T20:31:47.837218+0000","last_peered":"2024-09-15T20:31:47.837218+0000","last_clean":"2024-09-15T20:31:47.837218+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:31:47.837218+0000","last_undegraded":"2024-09-15T20:31:47.837218+0000","last_fullsized":"2024-09-15T20:31:47.837218+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:56.096771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.969706+0000","last_change":"2024-09-15T20:31:47.969706+0000","last_active":"2024-09-15T20:31:47.969706+0000","last_peered":"2024-09-15T20:31:47.969706+0000","last_clean":"2024-09-15T20:31:47.969706+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:31:47.969706+0000","last_undegraded":"2024-09-15T20:31:47.969706+0000","last_fullsized":"2024-09-15T20:31:47.969706+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:48.645009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":20,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.825141+0000","last_change":"2024-09-15T20:31:47.825565+0000","last_active":"2024-09-15T20:31:47.825141+0000","last_peered":"2024-09-15T20:31:47.825141+0000","last_clean":"2024-09-15T20:31:47.825141+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:31:47.825141+0000","last_undegraded":"2024-09-15T20:31:47.825141+0000","last_fullsized":"2024-09-15T20:31:47.825141+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:49:12.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"17'1","reported_seq":21,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.833645+0000","last_change":"2024-09-15T20:31:47.833645+0000","last_active":"2024-09-15T20:31:47.833645+0000","last_peered":"2024-09-15T20:31:47.833645+0000","last_clean":"2024-09-15T20:31:47.833645+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:31:47.833645+0000","last_undegraded":"2024-09-15T20:31:47.833645+0000","last_fullsized":"2024-09-15T20:31:47.833645+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:24.343248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"15'72","reported_seq":111,"reported_epoch":19,"state":"active+clean","last_fresh":"2024-09-15T20:31:47.824491+0000","last_change":"2024-09-15T20:31:40.658325+0000","last_active":"2024-09-15T20:31:47.824491+0000","last_peered":"2024-09-15T20:31:47.824491+0000","last_clean":"2024-09-15T20:31:47.824491+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:31:47.824491+0000","last_undegraded":"2024-09-15T20:31:47.824491+0000","last_fullsized":"2024-09-15T20:31:47.824491+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_clean_scrub_stamp":"2024-09-15T20:31:38.726961+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:41.743437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27196,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344644,"statfs":{"total":96636764160,"available":96608915456,"internally_reserved":0,"allocated":167936,"data_stored":52231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607556,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27192,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344648,"statfs":{"total":96636764160,"available":96608919552,"internally_reserved":0,"allocated":163840,"data_stored":52212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607557,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27188,"kb_used_data":160,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344652,"statfs":{"total":96636764160,"available":96608923648,"internally_reserved":0,"allocated":163840,"data_stored":52212,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607556,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":581140,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640262,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":225280,"data_stored":581140,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640261,"num_pgs":5,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27200,"kb_used_data":164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344640,"statfs":{"total":96636764160,"available":96608911360,"internally_reserved":0,"allocated":167936,"data_stored":52231,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:31:53.866 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-09-15T20:31:53.867 DEBUG:tasks.mon_thrash:Yielding 2024-09-15T20:31:53.867 INFO:teuthology.run_tasks:Running task workunit... 2024-09-15T20:31:53.882 INFO:tasks.workunit:Pulling workunits from ref 7498493279982dfe87f00616198a5967475f6169 2024-09-15T20:31:53.882 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-15T20:31:53.882 DEBUG:teuthology.orchestra.run.smithi146:> stat -- /home/ubuntu/cephtest/mnt.0 2024-09-15T20:31:53.883 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-09-15T20:31:53.883 INFO:tasks.mon_thrash.mon_thrasher:seed: 1726432313, revive delay: 90.0, thrash delay: 1.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-09-15T20:31:53.883 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:31:53.883 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:31:53.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:31:53.889 INFO:teuthology.orchestra.run.smithi146.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2024-09-15T20:31:53.889 DEBUG:teuthology.orchestra.run.smithi146:> mkdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T20:31:53.937 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2024-09-15T20:31:53.938 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2024-09-15T20:31:53.985 INFO:tasks.workunit:timeout=3h 2024-09-15T20:31:53.985 INFO:tasks.workunit:cleanup=True 2024-09-15T20:31:53.986 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7498493279982dfe87f00616198a5967475f6169 2024-09-15T20:31:53.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient: get_monmap_and_config 2024-09-15T20:31:53.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient: build_initial_monmap 2024-09-15T20:31:53.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient: monmap: 2024-09-15T20:31:53.999 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:53.999 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.000 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:53.998758+0000 2024-09-15T20:31:54.000 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:53.998758+0000 2024-09-15T20:31:54.000 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.000 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.000 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.000 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:54.000 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:54.000 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 1 Processor -- start 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 1 -- start start 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient: init 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient(hunting): picked mon.noname-b con 0x7fd428135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient(hunting): picked mon.noname-c con 0x7fd4280a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient(hunting): picked mon.noname-a con 0x7fd42813c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4280745e0 con 0x7fd42813c220 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42806ecf0 con 0x7fd4280a24e0 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42806efd0 con 0x7fd428135b60 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd42d513640 10 monclient(hunting): authenticate will time out at 1174.883495s 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd4277fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd42813c220 0x7fd42813e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51032/0 (socket says 172.21.15.78:51032) 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd4277fe640 1 -- 172.21.15.78:0/3017260026 learned_addr learned my addr 172.21.15.78:0/3017260026 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.995+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1513347793 0 0) 0x7fd42806efd0 con 0x7fd428135b60 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient(hunting): my global_id is 4412 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd408003530 con 0x7fd428135b60 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815139671 0 0) 0x7fd4280745e0 con 0x7fd42813c220 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient(hunting): my global_id is 4407 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd42806efd0 con 0x7fd42813c220 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4114366067 0 0) 0x7fd408003530 con 0x7fd428135b60 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 >> v1:172.21.15.146:6789/0 conn(0x7fd4280a24e0 legacy=0x7fd42813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 >> v1:172.21.15.78:6789/0 conn(0x7fd42813c220 legacy=0x7fd42813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 monclient: found mon.noname-b 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd428136650 con 0x7fd428135b60 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd4200016a0 con 0x7fd428135b60 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 5 monclient: authenticate success, global_id 4412 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: dump: 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.006 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd4200018c0 con 0x7fd428135b60 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 4 set_mon_vals no callback set 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 1 -- 172.21.15.78:0/3017260026 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd420001c90 con 0x7fd428135b60 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd425ffb640 10 monclient: dump: 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.007 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: shutdown 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- 172.21.15.78:0/3017260026 >> v1:172.21.15.78:6790/0 conn(0x7fd428135b60 legacy=0x7fd428135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- 172.21.15.78:0/3017260026 shutdown_connections 2024-09-15T20:31:54.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- 172.21.15.78:0/3017260026 >> 172.21.15.78:0/3017260026 conn(0x7fd4280a63c0 msgr2=0x7fd4280a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- 172.21.15.78:0/3017260026 shutdown_connections 2024-09-15T20:31:54.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- 172.21.15.78:0/3017260026 wait complete. 2024-09-15T20:31:54.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: monmap: 2024-09-15T20:31:54.013 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.013 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:54.013 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:54.013 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 librados: starting msgr at 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 librados: starting objecter 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 Processor -- start 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- start start 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 librados: setting wanted keys 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 librados: calling monclient init 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: init 2024-09-15T20:31:54.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient(hunting): picked mon.noname-a con 0x7fd4280a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient(hunting): picked mon.noname-b con 0x7fd428135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient(hunting): picked mon.noname-c con 0x7fd42813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42811cf80 con 0x7fd4280a24e0 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd4281390b0 con 0x7fd428135b60 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd42809d180 con 0x7fd42813c220 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient(hunting): authenticate will time out at 1174.886893s 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd426ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd4280a24e0 0x7fd428138290 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51034/0 (socket says 172.21.15.78:51034) 2024-09-15T20:31:54.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd426ffd640 1 -- 172.21.15.78:0/3346407447 learned_addr learned my addr 172.21.15.78:0/3346407447 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 client.?.objecter ms_handle_connect 0x7fd42813c220 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 client.?.objecter ms_handle_connect 0x7fd4280a24e0 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4229930567 0 0) 0x7fd42809d180 con 0x7fd42813c220 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): my global_id is 4415 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd40c004ec0 con 0x7fd42813c220 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2134181655 0 0) 0x7fd428136650 con 0x7fd4280a24e0 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): my global_id is 4410 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd42809d180 con 0x7fd4280a24e0 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 client.?.objecter ms_handle_connect 0x7fd428135b60 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1293613784 0 0) 0x7fd40c004ec0 con 0x7fd42813c220 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd428136650 con 0x7fd42813c220 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd418001940 con 0x7fd42813c220 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): dump: 2024-09-15T20:31:54.017 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 218602556 0 0) 0x7fd42809d180 con 0x7fd4280a24e0 2024-09-15T20:31:54.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd40c004ec0 con 0x7fd4280a24e0 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2105807048 0 0) 0x7fd4281390b0 con 0x7fd428135b60 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): my global_id is 4411 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd42809d180 con 0x7fd428135b60 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd420001c10 con 0x7fd4280a24e0 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient(hunting): dump: 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.019 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 475927640 0 0) 0x7fd428136650 con 0x7fd42813c220 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 >> v1:172.21.15.146:6789/0 conn(0x7fd428135b60 legacy=0x7fd4281389a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 >> v1:172.21.15.78:6789/0 conn(0x7fd4280a24e0 legacy=0x7fd428138290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 monclient: found mon.c 2024-09-15T20:31:54.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd42813b030 con 0x7fd42813c220 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 5 monclient: authenticate success, global_id 4415 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: _renew_subs 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd428139b40 con 0x7fd42813c220 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 client.4415.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: _renew_subs 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd428135530 con 0x7fd42813c220 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd42d513640 1 librados: init done 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd4180047f0 con 0x7fd42813c220 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd418004b40 con 0x7fd42813c220 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd41f7fe640 10 monclient: dump: 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.022 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:53.999+0000 7fd4257fa640 4 set_mon_vals no callback set 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd42d513640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd42d513640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd42d513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd400005090 con 0x7fd42813c220 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd418010bb0 con 0x7fd42813c220 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7fd41804ae80 con 0x7fd42813c220 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd41f7fe640 10 client.4415.objecter ms_dispatch 0x7fd428000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:54.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd41f7fe640 3 client.4415.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:54.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd41f7fe640 3 client.4415.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:54.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd41f7fe640 20 client.4415.objecter dump_active .. 0 homeless 2024-09-15T20:31:54.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd426ffd640 10 monclient: get_auth_request con 0x7fd40c03e330 auth_method 0 2024-09-15T20:31:54.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.003+0000 7fd41f7fe640 10 client.4415.objecter ms_handle_connect 0x7fd40c03e330 2024-09-15T20:31:54.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.007+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd418016920 con 0x7fd42813c220 2024-09-15T20:31:54.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.007+0000 7fd41f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.007+0000 7fd41f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:54.035 INFO:tasks.workunit.client.0.smithi146.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-09-15T20:31:54.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.231+0000 7fd42d513640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:31:54.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.231+0000 7fd42d513640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:31:54.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.231+0000 7fd42d513640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.231+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd400005380 con 0x7fd42813c220 2024-09-15T20:31:54.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.231+0000 7fd41f7fe640 1 -- 172.21.15.78:0/3346407447 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 312399330) 0x7fd418016220 con 0x7fd42813c220 2024-09-15T20:31:54.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.231+0000 7fd41f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:31:54.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.231+0000 7fd41f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:54.237 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":26,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:31:54.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 10 librados: watch_flush enter 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 10 librados: watch_flush exit 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 20 client.4415.objecter shutdown clearing up homeless session... 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 10 client.4415.objecter successfully canceled tick 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 >> v1:172.21.15.146:6812/435598642 conn(0x7fd40c03e330 legacy=0x7fd40c0407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 10 monclient: shutdown 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 >> v1:172.21.15.78:6790/0 conn(0x7fd42813c220 legacy=0x7fd4281d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 shutdown_connections 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 >> 172.21.15.78:0/3346407447 conn(0x7fd4280a63c0 msgr2=0x7fd4280a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 shutdown_connections 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 1 -- 172.21.15.78:0/3346407447 wait complete. 2024-09-15T20:31:54.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.235+0000 7fd42d513640 1 librados: shutdown 2024-09-15T20:31:54.254 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:31:54.254 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:31:54.254 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:31:54.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:31:54.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient: get_monmap_and_config 2024-09-15T20:31:54.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient: monmap: 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:54.361864+0000 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:54.361864+0000 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:54.363 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 Processor -- start 2024-09-15T20:31:54.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 -- start start 2024-09-15T20:31:54.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient: init 2024-09-15T20:31:54.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:54.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient(hunting): picked mon.noname-c con 0x7f17c0135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient(hunting): picked mon.noname-a con 0x7f17c00a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient(hunting): picked mon.noname-b con 0x7f17c013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c00745e0 con 0x7f17c00a24e0 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c006ecf0 con 0x7f17c0135b60 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c006efd0 con 0x7f17c013c220 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient(hunting): authenticate will time out at 1175.246445s 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c672a640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f17c013c220 0x7f17c013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54860/0 (socket says 172.21.15.78:54860) 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c672a640 1 -- 172.21.15.78:0/1839104760 learned_addr learned my addr 172.21.15.78:0/1839104760 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c5728640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f17c00a24e0 0x7f17c013ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51038/0 (socket says 172.21.15.78:51038) 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1813891575 0 0) 0x7f17c00745e0 con 0x7f17c00a24e0 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient(hunting): my global_id is 4413 2024-09-15T20:31:54.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17ac003530 con 0x7f17c00a24e0 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4165817840 0 0) 0x7f17c006efd0 con 0x7f17c013c220 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient(hunting): my global_id is 4418 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17c00745e0 con 0x7f17c013c220 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2182474725 0 0) 0x7f17ac003530 con 0x7f17c00a24e0 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 >> v1:172.21.15.78:6790/0 conn(0x7f17c013c220 legacy=0x7f17c013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 >> v1:172.21.15.146:6789/0 conn(0x7f17c0135b60 legacy=0x7f17c0135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 monclient: found mon.noname-a 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c0136650 con 0x7f17c00a24e0 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 5 monclient: authenticate success, global_id 4413 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17b00016a0 con 0x7f17c00a24e0 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: dump: 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.373 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17b00018c0 con 0x7f17c00a24e0 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 1 -- 172.21.15.78:0/1839104760 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17b0001c90 con 0x7f17c00a24e0 2024-09-15T20:31:54.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 4 set_mon_vals no callback set 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c4f27640 10 monclient: dump: 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.375 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 10 monclient: shutdown 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 -- 172.21.15.78:0/1839104760 >> v1:172.21.15.78:6789/0 conn(0x7f17c00a24e0 legacy=0x7f17c013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 -- 172.21.15.78:0/1839104760 shutdown_connections 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 -- 172.21.15.78:0/1839104760 >> 172.21.15.78:0/1839104760 conn(0x7f17c00a63c0 msgr2=0x7f17c00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.359+0000 7f17c818d640 1 -- 172.21.15.78:0/1839104760 shutdown_connections 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 -- 172.21.15.78:0/1839104760 wait complete. 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: monmap: 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:54.376 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 librados: starting msgr at 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 librados: starting objecter 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 Processor -- start 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 -- start start 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 librados: setting wanted keys 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 librados: calling monclient init 2024-09-15T20:31:54.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: init 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient(hunting): picked mon.noname-a con 0x7f17c00a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient(hunting): picked mon.noname-c con 0x7f17c0135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient(hunting): picked mon.noname-b con 0x7f17c013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c011cb30 con 0x7f17c00a24e0 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c0138f70 con 0x7f17c013c220 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f17c013ba40 con 0x7f17c0135b60 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient(hunting): authenticate will time out at 1175.249319s 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c5f29640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f17c00a24e0 0x7f17c0138150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51042/0 (socket says 172.21.15.78:51042) 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c5728640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f17c0135b60 0x7f17c0138860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54862/0 (socket says 172.21.15.78:54862) 2024-09-15T20:31:54.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c5f29640 1 -- 172.21.15.78:0/2998186584 learned_addr learned my addr 172.21.15.78:0/2998186584 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.?.objecter ms_handle_connect 0x7f17c00a24e0 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.?.objecter ms_handle_connect 0x7f17c0135b60 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 238013060 0 0) 0x7f17c013ba40 con 0x7f17c0135b60 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): my global_id is 4421 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17a4004ec0 con 0x7f17c0135b60 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692578095 0 0) 0x7f17c011cb30 con 0x7f17c00a24e0 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): my global_id is 4416 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17c013ba40 con 0x7f17c00a24e0 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.?.objecter ms_handle_connect 0x7f17c013c220 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 791091708 0 0) 0x7f17a4004ec0 con 0x7f17c0135b60 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17c011cb30 con 0x7f17c0135b60 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17b0001c10 con 0x7f17c0135b60 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): dump: 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.380 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3147937972 0 0) 0x7f17c013ba40 con 0x7f17c00a24e0 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17a4004ec0 con 0x7f17c00a24e0 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17bc001980 con 0x7f17c00a24e0 2024-09-15T20:31:54.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): dump: 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3198901817 0 0) 0x7f17c011cb30 con 0x7f17c0135b60 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 >> v1:172.21.15.146:6789/0 conn(0x7f17c013c220 legacy=0x7f17c01d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 >> v1:172.21.15.78:6789/0 conn(0x7f17c00a24e0 legacy=0x7f17c0138150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 monclient: found mon.c 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c009d180 con 0x7f17c0135b60 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 5 monclient: authenticate success, global_id 4421 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: _renew_subs 2024-09-15T20:31:54.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f17c013b100 con 0x7f17c0135b60 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 client.4421.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: _renew_subs 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f17c0135500 con 0x7f17c0135b60 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 librados: init done 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17b0001820 con 0x7f17c0135b60 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17b0004ce0 con 0x7f17c0135b60 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b7fff640 4 set_mon_vals no callback set 2024-09-15T20:31:54.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: dump: 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f17a80065d0 con 0x7f17a8002870 2024-09-15T20:31:54.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.4421.objecter ms_handle_connect 0x7f17a8002870 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.4421.objecter resend_mon_ops 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f17b0010d50 con 0x7f17c0135b60 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7f17b004afa0 con 0x7f17c0135b60 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 client.4421.objecter ms_dispatch 0x7f17c0000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 3 client.4421.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 3 client.4421.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 20 client.4421.objecter dump_active .. 0 homeless 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1782831043 0 0) 0x7f17a80065d0 con 0x7f17a8002870 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient(hunting): my global_id is 4419 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17c011cb30 con 0x7f17a8002870 2024-09-15T20:31:54.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17c672a640 10 monclient: get_auth_request con 0x7f17a403def0 auth_method 0 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 329541818 0 0) 0x7f17c011cb30 con 0x7f17a8002870 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17a80065d0 con 0x7f17a8002870 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17bc003a70 con 0x7f17a8002870 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 10 monclient: dump: 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.387 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1712380895 0 0) 0x7f17a80065d0 con 0x7f17a8002870 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.363+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17a800a170 con 0x7f17a8002870 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.367+0000 7f17b67fc640 10 client.4421.objecter ms_handle_connect 0x7f17a403def0 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.367+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f17bc0057d0 con 0x7f17a8002870 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.367+0000 7f17b67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.367+0000 7f17b67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.367+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 >> v1:172.21.15.78:6789/0 conn(0x7f17a8002870 legacy=0x7f17a8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.367+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 >> v1:172.21.15.78:6789/0 conn(0x7f17a8002870 legacy=0x7f17a8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f17a8009e60 con 0x7f17a8009120 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 client.4421.objecter ms_handle_connect 0x7f17a8009120 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 client.4421.objecter resend_mon_ops 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3560793404 0 0) 0x7f17a8009e60 con 0x7f17a8009120 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient(hunting): my global_id is 4422 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17a80065d0 con 0x7f17a8009120 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 844870873 0 0) 0x7f17a80065d0 con 0x7f17a8009120 2024-09-15T20:31:54.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17a8009e60 con 0x7f17a8009120 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17bc003fd0 con 0x7f17a8009120 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient: dump: 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.391 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2134819295 0 0) 0x7f17a8009e60 con 0x7f17a8009120 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17a800c400 con 0x7f17a8009120 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3972 (unknown 0 0 2206017251) 0x7f17bc003d70 con 0x7f17a8009120 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 >> v1:172.21.15.78:6789/0 conn(0x7f17a8009120 legacy=0x7f17a80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17b67fc640 1 -- 172.21.15.78:0/2998186584 >> v1:172.21.15.78:6789/0 conn(0x7f17a8009120 legacy=0x7f17a80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:31:54.392 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 4, 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 26923, 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 26, 2024-09-15T20:31:54.393 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.394 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.395 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:31:54.396 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.397 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:31:54.398 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.399 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.400 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.401 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:31:54.402 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.404 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:31:54.404 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:31:54.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 10 librados: watch_flush enter 2024-09-15T20:31:54.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 10 librados: watch_flush exit 2024-09-15T20:31:54.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 20 client.4421.objecter shutdown clearing up homeless session... 2024-09-15T20:31:54.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 10 client.4421.objecter successfully canceled tick 2024-09-15T20:31:54.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 >> v1:172.21.15.146:6812/435598642 conn(0x7f17a403def0 legacy=0x7f17a40403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 10 monclient: shutdown 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 >> v1:172.21.15.78:6790/0 conn(0x7f17c0135b60 legacy=0x7f17c0138860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c672a640 1 -- 172.21.15.78:0/2998186584 reap_dead start 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c672a640 1 -- 172.21.15.78:0/2998186584 reap_dead start 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 shutdown_connections 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 >> 172.21.15.78:0/2998186584 conn(0x7f17c00a63c0 msgr2=0x7f17c013f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 shutdown_connections 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 1 -- 172.21.15.78:0/2998186584 wait complete. 2024-09-15T20:31:54.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.371+0000 7f17c818d640 1 librados: shutdown 2024-09-15T20:31:54.407 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:31:54.407 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:31:54.407 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:31:54.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.519+0000 7fe87a2a1640 10 monclient: get_monmap_and_config 2024-09-15T20:31:54.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.519+0000 7fe87a2a1640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.519+0000 7fe87a2a1640 10 monclient: monmap: 2024-09-15T20:31:54.524 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.525 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.525 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:54.523809+0000 2024-09-15T20:31:54.525 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:54.523809+0000 2024-09-15T20:31:54.525 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.525 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.525 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.525 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:54.525 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:54.526 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.519+0000 7fe87a2a1640 1 Processor -- start 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.519+0000 7fe87a2a1640 1 -- start start 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.519+0000 7fe87a2a1640 10 monclient: init 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): picked mon.noname-b con 0x7fe874137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): picked mon.noname-a con 0x7fe874074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): picked mon.noname-c con 0x7fe87413e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe874138850 con 0x7fe874074500 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8740a5da0 con 0x7fe87413e370 2024-09-15T20:31:54.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe874073560 con 0x7fe874137d60 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): authenticate will time out at 1175.408333s 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe8737fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe874074500 0x7fe87413cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51060/0 (socket says 172.21.15.78:51060) 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe8737fe640 1 -- 172.21.15.78:0/3054861326 learned_addr learned my addr 172.21.15.78:0/3054861326 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe873fff640 1 --1- 172.21.15.78:0/3054861326 >> v1:172.21.15.78:6790/0 conn(0x7fe874137d60 0x7fe874138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54866/0 (socket says 172.21.15.78:54866) 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2907938945 0 0) 0x7fe874073560 con 0x7fe874137d60 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient(hunting): my global_id is 4424 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe85c003530 con 0x7fe874137d60 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3854074460 0 0) 0x7fe874138850 con 0x7fe874074500 2024-09-15T20:31:54.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient(hunting): my global_id is 4425 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe874073560 con 0x7fe874074500 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3908885284 0 0) 0x7fe874073560 con 0x7fe874074500 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 >> v1:172.21.15.78:6790/0 conn(0x7fe874137d60 legacy=0x7fe874138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 >> v1:172.21.15.146:6789/0 conn(0x7fe87413e370 legacy=0x7fe874140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 monclient: found mon.noname-a 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe874073840 con 0x7fe874074500 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe868001680 con 0x7fe874074500 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: dump: 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.536 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 5 monclient: authenticate success, global_id 4425 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe868001820 con 0x7fe874074500 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 1 -- 172.21.15.78:0/3054861326 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe868001c30 con 0x7fe874074500 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe872ffd640 10 monclient: dump: 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.537 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 4 set_mon_vals no callback set 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: shutdown 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3054861326 >> v1:172.21.15.78:6789/0 conn(0x7fe874074500 legacy=0x7fe87413cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3054861326 shutdown_connections 2024-09-15T20:31:54.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3054861326 >> 172.21.15.78:0/3054861326 conn(0x7fe8740aa650 msgr2=0x7fe8740aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3054861326 shutdown_connections 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3054861326 wait complete. 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: monmap: 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:54.539 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 librados: starting msgr at 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 librados: starting objecter 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 Processor -- start 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- start start 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 librados: setting wanted keys 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 librados: calling monclient init 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: init 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): picked mon.noname-a con 0x7fe874074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): picked mon.noname-c con 0x7fe874137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): picked mon.noname-b con 0x7fe87413e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe874132b20 con 0x7fe874074500 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8741405c0 con 0x7fe87413e370 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe8740a3fd0 con 0x7fe874137d60 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe87a2a1640 10 monclient(hunting): authenticate will time out at 1175.411441s 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe8737fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe874137d60 0x7fe87413feb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54870/0 (socket says 172.21.15.78:54870) 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe8737fe640 1 -- 172.21.15.78:0/3158852128 learned_addr learned my addr 172.21.15.78:0/3158852128 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe873fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe874074500 0x7fe87413f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51062/0 (socket says 172.21.15.78:51062) 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 client.?.objecter ms_handle_connect 0x7fe874137d60 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 client.?.objecter ms_handle_connect 0x7fe874074500 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1807942862 0 0) 0x7fe8740a3fd0 con 0x7fe874137d60 2024-09-15T20:31:54.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 monclient(hunting): my global_id is 4427 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe860004ec0 con 0x7fe874137d60 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 22843631 0 0) 0x7fe874132b20 con 0x7fe874074500 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 monclient(hunting): my global_id is 4428 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8740a3fd0 con 0x7fe874074500 2024-09-15T20:31:54.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 client.?.objecter ms_handle_connect 0x7fe87413e370 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.523+0000 7fe870ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2449899413 0 0) 0x7fe860004ec0 con 0x7fe874137d60 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe874132b20 con 0x7fe874137d60 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe868001c30 con 0x7fe874137d60 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): dump: 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.543 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3550159349 0 0) 0x7fe8740a3fd0 con 0x7fe874074500 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe860004ec0 con 0x7fe874074500 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe8640012e0 con 0x7fe874074500 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): dump: 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.544 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1136134965 0 0) 0x7fe874132b20 con 0x7fe874137d60 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 >> v1:172.21.15.146:6789/0 conn(0x7fe87413e370 legacy=0x7fe8741d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 >> v1:172.21.15.78:6789/0 conn(0x7fe874074500 legacy=0x7fe87413f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 monclient: found mon.c 2024-09-15T20:31:54.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8741418a0 con 0x7fe874137d60 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 5 monclient: authenticate success, global_id 4427 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 10 monclient: _renew_subs 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe874132840 con 0x7fe874137d60 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 10 client.4427.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 10 monclient: _renew_subs 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.546 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe8740a05c0 con 0x7fe874137d60 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 1 librados: init done 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe868001800 con 0x7fe874137d60 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe868004da0 con 0x7fe874137d60 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe8727fc640 4 set_mon_vals no callback set 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: dump: 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.547 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe8540065d0 con 0x7fe854002870 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 client.4427.objecter ms_handle_connect 0x7fe854002870 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 client.4427.objecter resend_mon_ops 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fe868010e10 con 0x7fe874137d60 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7fe86804b080 con 0x7fe874137d60 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 client.4427.objecter ms_dispatch 0x7fe874000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 3 client.4427.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:54.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 3 client.4427.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 20 client.4427.objecter dump_active .. 0 homeless 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3796135964 0 0) 0x7fe8540065d0 con 0x7fe854002870 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient(hunting): my global_id is 4430 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe874132b20 con 0x7fe854002870 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1166179904 0 0) 0x7fe874132b20 con 0x7fe854002870 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe8540065d0 con 0x7fe854002870 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe8640047f0 con 0x7fe854002870 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: dump: 2024-09-15T20:31:54.549 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe87883e640 10 monclient: get_auth_request con 0x7fe86003e0b0 auth_method 0 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3300325050 0 0) 0x7fe8540065d0 con 0x7fe854002870 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe85400a170 con 0x7fe854002870 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 client.4427.objecter ms_handle_connect 0x7fe86003e0b0 2024-09-15T20:31:54.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fe8640048f0 con 0x7fe854002870 2024-09-15T20:31:54.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:54.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 >> v1:172.21.15.78:6790/0 conn(0x7fe854002870 legacy=0x7fe854004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.527+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 >> v1:172.21.15.78:6790/0 conn(0x7fe854002870 legacy=0x7fe854004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:31:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:31:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fe854009e30 con 0x7fe8540090f0 2024-09-15T20:31:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 client.4427.objecter ms_handle_connect 0x7fe8540090f0 2024-09-15T20:31:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 client.4427.objecter resend_mon_ops 2024-09-15T20:31:54.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2561509849 0 0) 0x7fe854009e30 con 0x7fe8540090f0 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient(hunting): my global_id is 4433 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe8540065d0 con 0x7fe8540090f0 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4099592823 0 0) 0x7fe8540065d0 con 0x7fe8540090f0 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe854009e30 con 0x7fe8540090f0 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe864004f80 con 0x7fe8540090f0 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient: dump: 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.553 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 784329884 0 0) 0x7fe854009e30 con 0x7fe8540090f0 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe85400c400 con 0x7fe8540090f0 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4137 (unknown 0 0 2230547789) 0x7fe864006470 con 0x7fe8540090f0 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:54.554 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 >> v1:172.21.15.78:6790/0 conn(0x7fe8540090f0 legacy=0x7fe854009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe870ff9640 1 -- 172.21.15.78:0/3158852128 >> v1:172.21.15.78:6790/0 conn(0x7fe8540090f0 legacy=0x7fe854009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 4, 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 27087, 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:31:54.555 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 26, 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.556 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.557 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:31:54.558 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.559 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:31:54.561 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:31:54.563 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:31:54.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:31:54.565 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.566 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:31:54.566 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:31:54.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 10 librados: watch_flush enter 2024-09-15T20:31:54.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 10 librados: watch_flush exit 2024-09-15T20:31:54.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 20 client.4427.objecter shutdown clearing up homeless session... 2024-09-15T20:31:54.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 10 client.4427.objecter successfully canceled tick 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 >> v1:172.21.15.146:6812/435598642 conn(0x7fe86003e0b0 legacy=0x7fe860040570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 10 monclient: shutdown 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 >> v1:172.21.15.78:6790/0 conn(0x7fe874137d60 legacy=0x7fe87413feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87883e640 1 -- 172.21.15.78:0/3158852128 reap_dead start 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87883e640 1 -- 172.21.15.78:0/3158852128 reap_dead start 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 shutdown_connections 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 >> 172.21.15.78:0/3158852128 conn(0x7fe8740aa650 msgr2=0x7fe8740a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 shutdown_connections 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 1 -- 172.21.15.78:0/3158852128 wait complete. 2024-09-15T20:31:54.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.535+0000 7fe87a2a1640 1 librados: shutdown 2024-09-15T20:31:54.569 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:31:54.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:31:54.569 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:31:54.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 10 monclient: get_monmap_and_config 2024-09-15T20:31:54.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 10 monclient: monmap: 2024-09-15T20:31:54.684 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:54.683685+0000 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:54.683685+0000 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:54.685 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 1 Processor -- start 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 1 -- start start 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 10 monclient: init 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.679+0000 7fbf9e6e6640 10 monclient(hunting): picked mon.noname-c con 0x7fbf98137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): picked mon.noname-a con 0x7fbf98074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): picked mon.noname-b con 0x7fbf9813e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf98138850 con 0x7fbf98074500 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf980a5da0 con 0x7fbf98137d60 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf98073560 con 0x7fbf9813e370 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): authenticate will time out at 1175.568056s 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9cc83640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fbf9813e370 0x7fbf98140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:54904/0 (socket says 172.21.15.78:54904) 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9cc83640 1 -- 172.21.15.78:0/2121686994 learned_addr learned my addr 172.21.15.78:0/2121686994 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1856558246 0 0) 0x7fbf98073560 con 0x7fbf9813e370 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient(hunting): my global_id is 4436 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf84003530 con 0x7fbf9813e370 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1215510069 0 0) 0x7fbf98138850 con 0x7fbf98074500 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient(hunting): my global_id is 4431 2024-09-15T20:31:54.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf98073560 con 0x7fbf98074500 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2240110303 0 0) 0x7fbf84003530 con 0x7fbf9813e370 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 >> v1:172.21.15.146:6789/0 conn(0x7fbf98137d60 legacy=0x7fbf98138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 >> v1:172.21.15.78:6789/0 conn(0x7fbf98074500 legacy=0x7fbf9813cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 monclient: found mon.noname-b 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf98073840 con 0x7fbf9813e370 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf90001a20 con 0x7fbf9813e370 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: dump: 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 5 monclient: authenticate success, global_id 4436 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:54.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf90001c40 con 0x7fbf9813e370 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 4 set_mon_vals no callback set 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 1 -- 172.21.15.78:0/2121686994 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf90002010 con 0x7fbf9813e370 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf977fe640 10 monclient: dump: 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.697 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient: shutdown 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/2121686994 >> v1:172.21.15.78:6790/0 conn(0x7fbf9813e370 legacy=0x7fbf98140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/2121686994 shutdown_connections 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/2121686994 >> 172.21.15.78:0/2121686994 conn(0x7fbf980aa650 msgr2=0x7fbf980aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/2121686994 shutdown_connections 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/2121686994 wait complete. 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient: monmap: 2024-09-15T20:31:54.698 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 librados: starting msgr at 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 librados: starting objecter 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 Processor -- start 2024-09-15T20:31:54.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- start start 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 librados: setting wanted keys 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 librados: calling monclient init 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient: init 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): picked mon.noname-a con 0x7fbf98074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): picked mon.noname-b con 0x7fbf98137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): picked mon.noname-c con 0x7fbf9813e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf98139910 con 0x7fbf98074500 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf9809e0b0 con 0x7fbf98137d60 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbf9813a560 con 0x7fbf9813e370 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf9e6e6640 10 monclient(hunting): authenticate will time out at 1175.571139s 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf97fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbf98074500 0x7fbf98138af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51074/0 (socket says 172.21.15.78:51074) 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf97fff640 1 -- 172.21.15.78:0/977536914 learned_addr learned my addr 172.21.15.78:0/977536914 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 client.?.objecter ms_handle_connect 0x7fbf98074500 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 client.?.objecter ms_handle_connect 0x7fbf9813e370 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3620646206 0 0) 0x7fbf98139910 con 0x7fbf98074500 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): my global_id is 4434 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf78004ec0 con 0x7fbf98074500 2024-09-15T20:31:54.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2672604789 0 0) 0x7fbf9813a560 con 0x7fbf9813e370 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): my global_id is 4439 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf98139910 con 0x7fbf9813e370 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 client.?.objecter ms_handle_connect 0x7fbf98137d60 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3186872926 0 0) 0x7fbf78004ec0 con 0x7fbf98074500 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf9813a560 con 0x7fbf98074500 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf880019e0 con 0x7fbf98074500 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): dump: 2024-09-15T20:31:54.702 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2844471641 0 0) 0x7fbf98139910 con 0x7fbf9813e370 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf78004ec0 con 0x7fbf9813e370 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf90001df0 con 0x7fbf9813e370 2024-09-15T20:31:54.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): dump: 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2333565616 0 0) 0x7fbf9809e0b0 con 0x7fbf98137d60 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient(hunting): my global_id is 4429 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf98139910 con 0x7fbf98137d60 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1739600307 0 0) 0x7fbf9813a560 con 0x7fbf98074500 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 >> v1:172.21.15.78:6790/0 conn(0x7fbf9813e370 legacy=0x7fbf981d5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 >> v1:172.21.15.146:6789/0 conn(0x7fbf98137d60 legacy=0x7fbf98139200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 1 monclient: found mon.a 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.683+0000 7fbf957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9813c8c0 con 0x7fbf98074500 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 5 monclient: authenticate success, global_id 4434 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 10 monclient: _renew_subs 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9813c020 con 0x7fbf98074500 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 10 client.4434.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 10 monclient: _renew_subs 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbf9813b320 con 0x7fbf98074500 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 1 librados: init done 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbf88001ce0 con 0x7fbf98074500 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf88003a70 con 0x7fbf98074500 2024-09-15T20:31:54.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf96ffd640 4 set_mon_vals no callback set 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: dump: 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf7c0065d0 con 0x7fbf7c002870 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fbf88003c90 con 0x7fbf98074500 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 client.4434.objecter ms_handle_connect 0x7fbf7c002870 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 client.4434.objecter resend_mon_ops 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7fbf8804b190 con 0x7fbf98074500 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 client.4434.objecter ms_dispatch 0x7fbf98000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 3 client.4434.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 3 client.4434.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 20 client.4434.objecter dump_active .. 0 homeless 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3387005172 0 0) 0x7fbf7c0065d0 con 0x7fbf7c002870 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient(hunting): my global_id is 4432 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf9813a560 con 0x7fbf7c002870 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf9cc83640 10 monclient: get_auth_request con 0x7fbf7803e290 auth_method 0 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1708081361 0 0) 0x7fbf9813a560 con 0x7fbf7c002870 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf7c0065d0 con 0x7fbf7c002870 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf800017d0 con 0x7fbf7c002870 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: dump: 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.709 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 client.4434.objecter ms_handle_connect 0x7fbf7803e290 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3205832183 0 0) 0x7fbf7c0065d0 con 0x7fbf7c002870 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbf7c00a170 con 0x7fbf7c002870 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbf80001a50 con 0x7fbf7c002870 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 >> v1:172.21.15.146:6789/0 conn(0x7fbf7c002870 legacy=0x7fbf7c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.687+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 >> v1:172.21.15.146:6789/0 conn(0x7fbf7c002870 legacy=0x7fbf7c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbf7c009e30 con 0x7fbf7c0090f0 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 client.4434.objecter ms_handle_connect 0x7fbf7c0090f0 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 client.4434.objecter resend_mon_ops 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 674865033 0 0) 0x7fbf7c009e30 con 0x7fbf7c0090f0 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient(hunting): my global_id is 4435 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbf7c0065d0 con 0x7fbf7c0090f0 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2642873534 0 0) 0x7fbf7c0065d0 con 0x7fbf7c0090f0 2024-09-15T20:31:54.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbf7c009e30 con 0x7fbf7c0090f0 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbf800037a0 con 0x7fbf7c0090f0 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient: dump: 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.713 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2580238384 0 0) 0x7fbf7c009e30 con 0x7fbf7c0090f0 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbf7c00c400 con 0x7fbf7c0090f0 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4304 (unknown 0 0 614641353) 0x7fbf80003730 con 0x7fbf7c0090f0 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 >> v1:172.21.15.146:6789/0 conn(0x7fbf7c0090f0 legacy=0x7fbf7c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf957fa640 1 -- 172.21.15.78:0/977536914 >> v1:172.21.15.146:6789/0 conn(0x7fbf7c0090f0 legacy=0x7fbf7c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:31:54.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 4, 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 27203, 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 27, 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:31:54.715 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.716 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.717 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:31:54.718 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:31:54.719 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:31:54.720 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.721 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.723 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:31:54.725 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:31:54.726 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:31:54.727 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 10 librados: watch_flush enter 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 10 librados: watch_flush exit 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 20 client.4434.objecter shutdown clearing up homeless session... 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 10 client.4434.objecter successfully canceled tick 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 >> v1:172.21.15.146:6812/435598642 conn(0x7fbf7803e290 legacy=0x7fbf78040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 10 monclient: shutdown 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 >> v1:172.21.15.78:6789/0 conn(0x7fbf98074500 legacy=0x7fbf98138af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9cc83640 1 -- 172.21.15.78:0/977536914 reap_dead start 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9cc83640 1 -- 172.21.15.78:0/977536914 reap_dead start 2024-09-15T20:31:54.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 shutdown_connections 2024-09-15T20:31:54.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 >> 172.21.15.78:0/977536914 conn(0x7fbf980aa650 msgr2=0x7fbf980a8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 shutdown_connections 2024-09-15T20:31:54.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 1 -- 172.21.15.78:0/977536914 wait complete. 2024-09-15T20:31:54.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.695+0000 7fbf9e6e6640 1 librados: shutdown 2024-09-15T20:31:54.730 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:31:54.730 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T20:31:54.730 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T20:31:54.730 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T20:31:54.730 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T20:31:54.731 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T20:31:54.731 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:31:54.742 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T20:31:54.742 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:31:54.742 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:31:54.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient: get_monmap_and_config 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient: monmap: 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:54.890026+0000 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:54.890026+0000 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:54.891 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:54.892 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 1 Processor -- start 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 1 -- start start 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient: init 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient(hunting): picked mon.noname-b con 0x7f1a28135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient(hunting): picked mon.noname-a con 0x7f1a280a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a2813c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a280745e0 con 0x7f1a280a24e0 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2806ecf0 con 0x7f1a2813c220 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2806efd0 con 0x7f1a28135b60 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient(hunting): authenticate will time out at 1175.774592s 2024-09-15T20:31:54.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2c828640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f1a28135b60 legacy=0x7f1a28135f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a27fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1a280a24e0 0x7f1a2813ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51112/0 (socket says 172.21.15.78:51112) 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a27fff640 1 -- 172.21.15.78:0/1519333932 learned_addr learned my addr 172.21.15.78:0/1519333932 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1530587098 0 0) 0x7f1a280745e0 con 0x7f1a280a24e0 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient(hunting): my global_id is 4437 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a10003530 con 0x7f1a280a24e0 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 106286770 0 0) 0x7f1a10003530 con 0x7f1a280a24e0 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 >> v1:172.21.15.78:6790/0 conn(0x7f1a28135b60 legacy=0x7f1a28135f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 >> v1:172.21.15.146:6789/0 conn(0x7f1a2813c220 legacy=0x7f1a2813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 monclient: found mon.noname-a 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a28136650 con 0x7f1a280a24e0 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1a180016a0 con 0x7f1a280a24e0 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 5 monclient: authenticate success, global_id 4437 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: dump: 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a180018c0 con 0x7f1a280a24e0 2024-09-15T20:31:54.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a2ea8c640 4 set_mon_vals no callback set 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 1 -- 172.21.15.78:0/1519333932 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1a18001cd0 con 0x7f1a280a24e0 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.887+0000 7f1a277fe640 10 monclient: dump: 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.903 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: shutdown 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1519333932 >> v1:172.21.15.78:6789/0 conn(0x7f1a280a24e0 legacy=0x7f1a2813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1519333932 shutdown_connections 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1519333932 >> 172.21.15.78:0/1519333932 conn(0x7f1a280a63c0 msgr2=0x7f1a280a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1519333932 shutdown_connections 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1519333932 wait complete. 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: build_initial_monmap 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: monmap: 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:54.904 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 librados: starting msgr at 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 librados: starting objecter 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 Processor -- start 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- start start 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 librados: setting wanted keys 2024-09-15T20:31:54.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 librados: calling monclient init 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: init 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient(hunting): picked mon.noname-a con 0x7f1a280a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient(hunting): picked mon.noname-c con 0x7f1a28135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient(hunting): picked mon.noname-b con 0x7f1a2813c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2813bb20 con 0x7f1a280a24e0 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2809d180 con 0x7f1a2813c220 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a2809e120 con 0x7f1a28135b60 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a27fff640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f1a28135b60 legacy=0x7f1a280723c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient(hunting): authenticate will time out at 1175.777551s 2024-09-15T20:31:54.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2c828640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1a280a24e0 0x7f1a28071cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51116/0 (socket says 172.21.15.78:51116) 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2c828640 1 -- 172.21.15.78:0/1930691236 learned_addr learned my addr 172.21.15.78:0/1930691236 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 client.?.objecter ms_handle_connect 0x7f1a280a24e0 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2040943202 0 0) 0x7f1a1c003aa0 con 0x7f1a280a24e0 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient(hunting): _negotiate 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient(hunting): my global_id is 4440 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a14004ec0 con 0x7f1a280a24e0 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 client.?.objecter ms_handle_connect 0x7f1a2813c220 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4093737120 0 0) 0x7f1a14004ec0 con 0x7f1a280a24e0 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1a1c003aa0 con 0x7f1a280a24e0 2024-09-15T20:31:54.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1a1c0043b0 con 0x7f1a280a24e0 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient(hunting): dump: 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.908 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3906561801 0 0) 0x7f1a1c003aa0 con 0x7f1a280a24e0 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 >> v1:172.21.15.78:6790/0 conn(0x7f1a28135b60 legacy=0x7f1a280723c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 >> v1:172.21.15.146:6789/0 conn(0x7f1a2813c220 legacy=0x7f1a281d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient: _finish_hunting 0 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 monclient: found mon.a 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a2809e400 con 0x7f1a280a24e0 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient: _finish_auth 0 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient: _check_auth_tickets 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:54.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 5 monclient: authenticate success, global_id 4440 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: _renew_subs 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1a28072ad0 con 0x7f1a280a24e0 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: _renew_subs 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1a2813f9d0 con 0x7f1a280a24e0 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 librados: init done 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1a1c0045d0 con 0x7f1a280a24e0 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1a1c0049e0 con 0x7f1a280a24e0 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:54.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a26ffd640 4 set_mon_vals no callback set 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 monclient: dump: 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a08005090 con 0x7f1a280a24e0 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1a1c004b80 con 0x7f1a280a24e0 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7f1a1c04b300 con 0x7f1a280a24e0 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 10 client.4440.objecter ms_dispatch 0x7f1a28000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 3 client.4440.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 3 client.4440.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a257fa640 20 client.4440.objecter dump_active .. 0 homeless 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.891+0000 7f1a27fff640 10 monclient: get_auth_request con 0x7f1a1403df20 auth_method 0 2024-09-15T20:31:54.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.895+0000 7f1a257fa640 10 client.4440.objecter ms_handle_connect 0x7f1a1403df20 2024-09-15T20:31:54.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.895+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1a2813bb20 con 0x7f1a280a24e0 2024-09-15T20:31:54.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.895+0000 7f1a257fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:54.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:54.895+0000 7f1a257fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:55.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.127+0000 7f1a2ea8c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:31:55.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.127+0000 7f1a2ea8c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:31:55.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.127+0000 7f1a2ea8c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:55.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.127+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1a08005380 con 0x7f1a280a24e0 2024-09-15T20:31:55.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.127+0000 7f1a257fa640 1 -- 172.21.15.78:0/1930691236 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 4022134752) 0x7f1a1c016cc0 con 0x7f1a280a24e0 2024-09-15T20:31:55.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.127+0000 7f1a257fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:31:55.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.127+0000 7f1a257fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:55.132 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":27,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:31:55.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 10 librados: watch_flush enter 2024-09-15T20:31:55.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 10 librados: watch_flush exit 2024-09-15T20:31:55.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 20 client.4440.objecter shutdown clearing up homeless session... 2024-09-15T20:31:55.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 10 client.4440.objecter successfully canceled tick 2024-09-15T20:31:55.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 >> v1:172.21.15.146:6812/435598642 conn(0x7f1a1403df20 legacy=0x7f1a140403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:55.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 10 monclient: shutdown 2024-09-15T20:31:55.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:55.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 >> v1:172.21.15.78:6789/0 conn(0x7f1a280a24e0 legacy=0x7f1a28071cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:55.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 shutdown_connections 2024-09-15T20:31:55.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 >> 172.21.15.78:0/1930691236 conn(0x7f1a280a63c0 msgr2=0x7f1a2812e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:55.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 shutdown_connections 2024-09-15T20:31:55.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 1 -- 172.21.15.78:0/1930691236 wait complete. 2024-09-15T20:31:55.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:55.131+0000 7f1a2ea8c640 1 librados: shutdown 2024-09-15T20:31:58.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:31:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.275+0000 7f1bfb418640 10 monclient: get_monmap_and_config 2024-09-15T20:31:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.275+0000 7f1bfb418640 10 monclient: build_initial_monmap 2024-09-15T20:31:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.275+0000 7f1bfb418640 10 monclient: monmap: 2024-09-15T20:31:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:58.280 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:58.279524+0000 2024-09-15T20:31:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:58.279524+0000 2024-09-15T20:31:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:31:58.281 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:31:58.281 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.275+0000 7f1bfb418640 1 Processor -- start 2024-09-15T20:31:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.275+0000 7f1bfb418640 1 -- start start 2024-09-15T20:31:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.275+0000 7f1bfb418640 10 monclient: init 2024-09-15T20:31:58.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf40a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf40745e0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf406ecf0 con 0x7f1bf413c220 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bf91b4640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f1bf4135b60 legacy=0x7f1bf4135f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf406efd0 con 0x7f1bf4135b60 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): authenticate will time out at 1179.164263s 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bf89b3640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1bf40a24e0 0x7f1bf413ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51132/0 (socket says 172.21.15.78:51132) 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bf89b3640 1 -- 172.21.15.78:0/2146487248 learned_addr learned my addr 172.21.15.78:0/2146487248 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:58.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3074032144 0 0) 0x7f1bf40745e0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient(hunting): _negotiate 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient(hunting): my global_id is 4443 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1be0003530 con 0x7f1bf40a24e0 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3470015975 0 0) 0x7f1be0003530 con 0x7f1bf40a24e0 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 >> v1:172.21.15.78:6790/0 conn(0x7f1bf4135b60 legacy=0x7f1bf4135f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 >> v1:172.21.15.146:6789/0 conn(0x7f1bf413c220 legacy=0x7f1bf413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: _finish_hunting 0 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 monclient: found mon.noname-a 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:31:58.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf4136650 con 0x7f1bf40a24e0 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: _finish_auth 0 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: _check_auth_tickets 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1bf00016a0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 5 monclient: authenticate success, global_id 4443 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: dump: 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:58.287 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1bf00018c0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 1 -- 172.21.15.78:0/2146487248 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1bf0001cd0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: get_monmap_and_config success 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 4 set_mon_vals no callback set 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:58.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bebfff640 10 monclient: dump: 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: shutdown 2024-09-15T20:31:58.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- 172.21.15.78:0/2146487248 >> v1:172.21.15.78:6789/0 conn(0x7f1bf40a24e0 legacy=0x7f1bf413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- 172.21.15.78:0/2146487248 shutdown_connections 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- 172.21.15.78:0/2146487248 >> 172.21.15.78:0/2146487248 conn(0x7f1bf40a63c0 msgr2=0x7f1bf40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- 172.21.15.78:0/2146487248 shutdown_connections 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- 172.21.15.78:0/2146487248 wait complete. 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: build_initial_monmap 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: monmap: 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:58.290 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 librados: starting msgr at 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 librados: starting objecter 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 Processor -- start 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- start start 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 librados: setting wanted keys 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 librados: calling monclient init 2024-09-15T20:31:58.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: init 2024-09-15T20:31:58.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: _reopen_session rank -1 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): picked mon.noname-a con 0x7f1bf40a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): picked mon.noname-b con 0x7f1bf4135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): picked mon.noname-c con 0x7f1bf413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf4138930 con 0x7f1bf40a24e0 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf4138510 con 0x7f1bf4135b60 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1bf40728c0 con 0x7f1bf413c220 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): _renew_subs 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bfb418640 10 monclient(hunting): authenticate will time out at 1179.167860s 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bf99b5640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f1bf413c220 legacy=0x7f1bf41d2c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bf91b4640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1bf40a24e0 0x7f1bf4071aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:51148/0 (socket says 172.21.15.78:51148) 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.279+0000 7f1bf91b4640 1 -- 172.21.15.78:0/3862611052 learned_addr learned my addr 172.21.15.78:0/3862611052 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:31:58.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 client.?.objecter ms_handle_connect 0x7f1bf40a24e0 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3234767930 0 0) 0x7f1be4003aa0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient(hunting): my global_id is 4446 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1bd8004ef0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 client.?.objecter ms_handle_connect 0x7f1bf4135b60 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2792998505 0 0) 0x7f1bd8004ef0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1be4003aa0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1be40043b0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient(hunting): dump: 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:31:58.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3137640384 0 0) 0x7f1be4003aa0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 >> v1:172.21.15.78:6790/0 conn(0x7f1bf413c220 legacy=0x7f1bf41d2c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 >> v1:172.21.15.146:6789/0 conn(0x7f1bf4135b60 legacy=0x7f1bf40721b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 monclient: found mon.a 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bf409d180 con 0x7f1bf40a24e0 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 5 monclient: authenticate success, global_id 4446 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 10 monclient: _renew_subs 2024-09-15T20:31:58.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1bf4138230 con 0x7f1bf40a24e0 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 10 client.4446.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 10 monclient: _renew_subs 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1bf4136d10 con 0x7f1bf40a24e0 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 1 librados: init done 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1be40045d0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1be40049e0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1beb7fe640 4 set_mon_vals no callback set 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 monclient: dump: 2024-09-15T20:31:58.299 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:58.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd4005090 con 0x7f1bf40a24e0 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1be4004b80 con 0x7f1bf40a24e0 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7f1be404b2b0 con 0x7f1bf40a24e0 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 client.4446.objecter ms_dispatch 0x7f1bf4000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 3 client.4446.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 3 client.4446.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 20 client.4446.objecter dump_active .. 0 homeless 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1bf89b3640 10 monclient: get_auth_request con 0x7f1bd803df80 auth_method 0 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.283+0000 7f1be9ffb640 10 client.4446.objecter ms_handle_connect 0x7f1bd803df80 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.287+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1bf4138930 con 0x7f1bf40a24e0 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.287+0000 7f1be9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:31:58.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.287+0000 7f1be9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:31:58.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:31:58.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:31:58.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:31:58.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1bd4005380 con 0x7f1bf40a24e0 2024-09-15T20:31:58.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1be9ffb640 1 -- 172.21.15.78:0/3862611052 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 4132226761) 0x7f1be4016c70 con 0x7f1bf40a24e0 2024-09-15T20:31:58.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1be9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:31:58.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1be9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:31:58.527 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":30,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:31:58.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 10 librados: watch_flush enter 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 10 librados: watch_flush exit 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 20 client.4446.objecter shutdown clearing up homeless session... 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 10 client.4446.objecter successfully canceled tick 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 >> v1:172.21.15.146:6812/435598642 conn(0x7f1bd803df80 legacy=0x7f1bd8040440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 10 monclient: shutdown 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 >> v1:172.21.15.78:6789/0 conn(0x7f1bf40a24e0 legacy=0x7f1bf4071aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 shutdown_connections 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 >> 172.21.15.78:0/3862611052 conn(0x7f1bf40a63c0 msgr2=0x7f1bf40a5280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 shutdown_connections 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 1 -- 172.21.15.78:0/3862611052 wait complete. 2024-09-15T20:31:58.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:31:58.523+0000 7f1bfb418640 1 librados: shutdown 2024-09-15T20:32:01.545 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:32:01.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient: get_monmap_and_config 2024-09-15T20:32:01.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient: build_initial_monmap 2024-09-15T20:32:01.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient: monmap: 2024-09-15T20:32:01.674 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:32:01.673803+0000 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:32:01.673803+0000 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:32:01.675 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:01.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 1 Processor -- start 2024-09-15T20:32:01.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 1 -- start start 2024-09-15T20:32:01.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient: init 2024-09-15T20:32:01.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient: _reopen_session rank -1 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient(hunting): picked mon.noname-b con 0x7fc088137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient(hunting): picked mon.noname-a con 0x7fc088074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient(hunting): picked mon.noname-c con 0x7fc08813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc088138850 con 0x7fc088074500 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0880a5da0 con 0x7fc08813e370 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc088073560 con 0x7fc088137d60 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient(hunting): _renew_subs 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08ffe4640 10 monclient(hunting): authenticate will time out at 1182.558418s 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08dd80640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc088138140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08d57f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc088074500 0x7fc08813cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34668/0 (socket says 172.21.15.78:34668) 2024-09-15T20:32:01.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.671+0000 7fc08d57f640 1 -- 172.21.15.78:0/2804777688 learned_addr learned my addr 172.21.15.78:0/2804777688 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:32:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:01.875+0000 7fc08dd80640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc088138140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:02.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:02.279+0000 7fc08dd80640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc088138140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:03.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:03.083+0000 7fc08dd80640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc088138140 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 10 monclient(hunting): tick 2024-09-15T20:32:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 1 monclient(hunting): continuing hunt 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc088138140 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.146:6789/0 conn(0x7fc08813e370 legacy=0x7fc088140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6789/0 conn(0x7fc088074500 legacy=0x7fc08813cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc06c0036d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc08813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fc088137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc088073840 con 0x7fc06c0036d0 2024-09-15T20:32:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc088073560 con 0x7fc08813e370 2024-09-15T20:32:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06c0103d0 con 0x7fc088137d60 2024-09-15T20:32:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc08e581640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.671+0000 7fc07f7fe640 10 monclient(hunting): _renew_subs 2024-09-15T20:32:04.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:04.875+0000 7fc08e581640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:05.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:05.279+0000 7fc08e581640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:06.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:06.083+0000 7fc08e581640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:07.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 10 monclient(hunting): tick 2024-09-15T20:32:07.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:32:07.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 1 monclient(hunting): continuing hunt 2024-09-15T20:32:07.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:32:07.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:32:07.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.146:6789/0 conn(0x7fc08813e370 legacy=0x7fc06c0093b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:07.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6789/0 conn(0x7fc06c0036d0 legacy=0x7fc06c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:07.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:32:07.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 10 monclient(hunting): picked mon.noname-b con 0x7fc088137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:07.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 10 monclient(hunting): picked mon.noname-c con 0x7fc06c010680 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:07.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 10 monclient(hunting): picked mon.noname-a con 0x7fc06c0129f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:07.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06c011730 con 0x7fc06c0129f0 2024-09-15T20:32:07.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06c0103d0 con 0x7fc06c010680 2024-09-15T20:32:07.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 1 -- 172.21.15.78:0/2804777688 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc06c0073b0 con 0x7fc088137d60 2024-09-15T20:32:07.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc07f7fe640 10 monclient(hunting): _renew_subs 2024-09-15T20:32:07.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.671+0000 7fc08dd80640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:07.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:07.875+0000 7fc08dd80640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:08.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.279+0000 7fc08dd80640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:08.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1014627708 0 0) 0x7fc06c0103d0 con 0x7fc06c010680 2024-09-15T20:32:08.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient(hunting): _negotiate 2024-09-15T20:32:08.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:08.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:08.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient(hunting): my global_id is 4438 2024-09-15T20:32:08.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc074003530 con 0x7fc06c010680 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3258542175 0 0) 0x7fc074003530 con 0x7fc06c010680 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc06c00cbb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.78:6789/0 conn(0x7fc06c0129f0 legacy=0x7fc06c014de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient: _finish_hunting 0 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 1 monclient: found mon.noname-c 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc06c008630 con 0x7fc06c010680 2024-09-15T20:32:08.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient: _finish_auth 0 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient: _check_auth_tickets 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc084001560 con 0x7fc06c010680 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08cd7e640 10 monclient: dump: 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:08.911 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.903+0000 7fc08ffe4640 5 monclient: authenticate success, global_id 4438 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc084001700 con 0x7fc06c010680 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08cd7e640 10 monclient: handle_config config(0 keys) 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 10 monclient: get_monmap_and_config success 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 4 set_mon_vals no callback set 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08cd7e640 1 -- 172.21.15.78:0/2804777688 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc084001b90 con 0x7fc06c010680 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08cd7e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08cd7e640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08cd7e640 10 monclient: dump: 2024-09-15T20:32:08.912 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:08.913 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:08.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 10 monclient: shutdown 2024-09-15T20:32:08.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2804777688 >> v1:172.21.15.146:6789/0 conn(0x7fc06c010680 legacy=0x7fc06c010b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2804777688 shutdown_connections 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2804777688 >> 172.21.15.78:0/2804777688 conn(0x7fc0880aa650 msgr2=0x7fc0880aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2804777688 shutdown_connections 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2804777688 wait complete. 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 10 monclient: build_initial_monmap 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 10 monclient: monmap: 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:32:08.916 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 librados: starting msgr at 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 librados: starting objecter 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 Processor -- start 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 -- start start 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 librados: setting wanted keys 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 1 librados: calling monclient init 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 10 monclient: init 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 10 monclient: _reopen_session rank -1 2024-09-15T20:32:08.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.907+0000 7fc08ffe4640 10 monclient(hunting): picked mon.noname-a con 0x7fc06c0129f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08ffe4640 10 monclient(hunting): picked mon.noname-c con 0x7fc088137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08d57f640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc0881d1bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08ffe4640 10 monclient(hunting): picked mon.noname-b con 0x7fc08813c290 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08ffe4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc08813d4d0 con 0x7fc06c0129f0 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08ffe4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc08813d7b0 con 0x7fc08813c290 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08ffe4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0881cd760 con 0x7fc088137d60 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08ffe4640 10 monclient(hunting): _renew_subs 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08ffe4640 10 monclient(hunting): authenticate will time out at 1189.796220s 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08dd80640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc06c0129f0 0x7fc088134d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34702/0 (socket says 172.21.15.78:34702) 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc08dd80640 1 -- 172.21.15.78:0/2283588417 learned_addr learned my addr 172.21.15.78:0/2283588417 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 client.?.objecter ms_handle_connect 0x7fc06c0129f0 2024-09-15T20:32:08.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 client.?.objecter ms_handle_connect 0x7fc08813c290 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988660613 0 0) 0x7fc08813d7b0 con 0x7fc08813c290 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): my global_id is 4441 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc06c0103d0 con 0x7fc08813c290 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4146548568 0 0) 0x7fc08813d4d0 con 0x7fc06c0129f0 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): my global_id is 4452 2024-09-15T20:32:08.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc08813d7b0 con 0x7fc06c0129f0 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2213648411 0 0) 0x7fc06c0103d0 con 0x7fc08813c290 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc08813d4d0 con 0x7fc08813c290 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc080001be0 con 0x7fc08813c290 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): dump: 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:08.920 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:08.921 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:08.921 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:08.921 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:08.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:32:08.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1810133110 0 0) 0x7fc08813d7b0 con 0x7fc06c0129f0 2024-09-15T20:32:08.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc06c0103d0 con 0x7fc06c0129f0 2024-09-15T20:32:08.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc078004300 con 0x7fc06c0129f0 2024-09-15T20:32:08.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:32:08.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:08.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.911+0000 7fc07e7fc640 10 monclient(hunting): dump: 2024-09-15T20:32:08.923 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 87099532 0 0) 0x7fc06c0103d0 con 0x7fc06c0129f0 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 >> v1:172.21.15.78:6790/0 conn(0x7fc088137d60 legacy=0x7fc0881d1bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 >> v1:172.21.15.146:6789/0 conn(0x7fc08813c290 legacy=0x7fc0881d52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:08.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 1 monclient: found mon.a 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc08813dff0 con 0x7fc06c0129f0 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 10 monclient: _finish_auth 0 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 5 monclient: authenticate success, global_id 4452 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 10 monclient: _renew_subs 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc08813b730 con 0x7fc06c0129f0 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:32:08.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 10 monclient: _renew_subs 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc0881cc870 con 0x7fc06c0129f0 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 1 librados: init done 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc078004750 con 0x7fc06c0129f0 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc078004b60 con 0x7fc06c0129f0 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07e7fc640 10 monclient: dump: 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.926 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc07ffff640 4 set_mon_vals no callback set 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.915+0000 7fc08ffe4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc08ffe4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc08ffe4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc070005090 con 0x7fc06c0129f0 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fc078004d80 con 0x7fc06c0129f0 2024-09-15T20:32:08.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7fc07804b020 con 0x7fc06c0129f0 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc07e7fc640 10 client.4452.objecter ms_dispatch 0x7fc088000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc07e7fc640 3 client.4452.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc07e7fc640 3 client.4452.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc07e7fc640 20 client.4452.objecter dump_active .. 0 homeless 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc08d57f640 10 monclient: get_auth_request con 0x7fc06c03f490 auth_method 0 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.919+0000 7fc07e7fc640 10 client.4452.objecter ms_handle_connect 0x7fc06c03f490 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.923+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc078016be0 con 0x7fc06c0129f0 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.923+0000 7fc07e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:32:08.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:08.923+0000 7fc07e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:32:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:32:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:32:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc070005380 con 0x7fc06c0129f0 2024-09-15T20:32:09.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc07e7fc640 1 -- 172.21.15.78:0/2283588417 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1399 (unknown 254746657 0 3948400050) 0x7fc0780164e0 con 0x7fc06c0129f0 2024-09-15T20:32:09.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc07e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:32:09.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc07e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:32:09.167 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:32:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 10 librados: watch_flush enter 2024-09-15T20:32:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 10 librados: watch_flush exit 2024-09-15T20:32:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 20 client.4452.objecter shutdown clearing up homeless session... 2024-09-15T20:32:09.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 10 client.4452.objecter successfully canceled tick 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 >> v1:172.21.15.146:6812/435598642 conn(0x7fc06c03f490 legacy=0x7fc06c041950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 10 monclient: shutdown 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 >> v1:172.21.15.78:6789/0 conn(0x7fc06c0129f0 legacy=0x7fc088134d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 shutdown_connections 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 >> 172.21.15.78:0/2283588417 conn(0x7fc0880aa650 msgr2=0x7fc0880a8870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 shutdown_connections 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 1 -- 172.21.15.78:0/2283588417 wait complete. 2024-09-15T20:32:09.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.163+0000 7fc08ffe4640 1 librados: shutdown 2024-09-15T20:32:09.186 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:32:09.186 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T20:32:09.186 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:32:09.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.303+0000 7fcef5da8640 10 monclient: get_monmap_and_config 2024-09-15T20:32:09.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.303+0000 7fcef5da8640 10 monclient: build_initial_monmap 2024-09-15T20:32:09.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.303+0000 7fcef5da8640 10 monclient: monmap: 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:32:09.308686+0000 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:32:09.308686+0000 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:32:09.310 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 Processor -- start 2024-09-15T20:32:09.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- start start 2024-09-15T20:32:09.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient: init 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient: _reopen_session rank -1 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient(hunting): picked mon.noname-c con 0x7fcef0134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient(hunting): picked mon.noname-b con 0x7fcef009f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient(hunting): picked mon.noname-a con 0x7fcef006e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef00a0470 con 0x7fcef006e960 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef009feb0 con 0x7fcef0134bb0 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef009e4a0 con 0x7fcef009f5a0 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient(hunting): _renew_subs 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient(hunting): authenticate will time out at 1190.193529s 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceeeffd640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7fcef009f5a0 legacy=0x7fcef01399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceeffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcef006e960 0x7fcef013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34714/0 (socket says 172.21.15.78:34714) 2024-09-15T20:32:09.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceeffff640 1 -- 172.21.15.78:0/144949865 learned_addr learned my addr 172.21.15.78:0/144949865 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2360612032 0 0) 0x7fcee80014a0 con 0x7fcef006e960 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient(hunting): my global_id is 4455 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fced8005740 con 0x7fcef006e960 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3065367341 0 0) 0x7fced8005740 con 0x7fcef006e960 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 >> v1:172.21.15.78:6790/0 conn(0x7fcef009f5a0 legacy=0x7fcef01399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 >> v1:172.21.15.146:6789/0 conn(0x7fcef0134bb0 legacy=0x7fcef0134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 monclient: found mon.noname-a 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef00a0ff0 con 0x7fcef006e960 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: _finish_auth 0 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcee8001bf0 con 0x7fcef006e960 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: dump: 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.315 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 5 monclient: authenticate success, global_id 4455 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcee8001e10 con 0x7fcef006e960 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 1 -- 172.21.15.78:0/144949865 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcee8004dd0 con 0x7fcef006e960 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fceee7fc640 10 monclient: dump: 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.316 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient: get_monmap_and_config success 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 4 set_mon_vals no callback set 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient: shutdown 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- 172.21.15.78:0/144949865 >> v1:172.21.15.78:6789/0 conn(0x7fcef006e960 legacy=0x7fcef013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- 172.21.15.78:0/144949865 shutdown_connections 2024-09-15T20:32:09.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- 172.21.15.78:0/144949865 >> 172.21.15.78:0/144949865 conn(0x7fcef00a63c0 msgr2=0x7fcef00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:32:09.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- 172.21.15.78:0/144949865 shutdown_connections 2024-09-15T20:32:09.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 -- 172.21.15.78:0/144949865 wait complete. 2024-09-15T20:32:09.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient: build_initial_monmap 2024-09-15T20:32:09.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 10 monclient: monmap: 2024-09-15T20:32:09.322 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:32:09.322 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:32:09.322 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:32:09.322 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 librados: starting msgr at 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.307+0000 7fcef5da8640 1 librados: starting objecter 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 Processor -- start 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 -- start start 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 librados: setting wanted keys 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 librados: calling monclient init 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient: init 2024-09-15T20:32:09.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient: _reopen_session rank -1 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient(hunting): picked mon.noname-b con 0x7fcef006e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient(hunting): picked mon.noname-a con 0x7fcef009f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient(hunting): picked mon.noname-c con 0x7fcef0134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef0128360 con 0x7fcef009f5a0 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef0126e40 con 0x7fcef006e960 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcef0128640 con 0x7fcef0134bb0 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient(hunting): _renew_subs 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient(hunting): authenticate will time out at 1190.196587s 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fceeffff640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7fcef0134bb0 legacy=0x7fcef01d4c50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fceeeffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcef009f5a0 0x7fcef0133c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34718/0 (socket says 172.21.15.78:34718) 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fceeeffd640 1 -- 172.21.15.78:0/4153750773 learned_addr learned my addr 172.21.15.78:0/4153750773 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:32:09.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 client.?.objecter ms_handle_connect 0x7fcef009f5a0 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3863040843 0 0) 0x7fcef0128360 con 0x7fcef009f5a0 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): my global_id is 4458 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcedc004ef0 con 0x7fcef009f5a0 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 client.?.objecter ms_handle_connect 0x7fcef006e960 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1882504237 0 0) 0x7fcedc004ef0 con 0x7fcef009f5a0 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcef0128360 con 0x7fcef009f5a0 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcee40041b0 con 0x7fcef009f5a0 2024-09-15T20:32:09.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): dump: 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2661337115 0 0) 0x7fcef0126e40 con 0x7fcef006e960 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): my global_id is 4447 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcedc004ef0 con 0x7fcef006e960 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 511835698 0 0) 0x7fcef0128360 con 0x7fcef009f5a0 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 >> v1:172.21.15.78:6790/0 conn(0x7fcef0134bb0 legacy=0x7fcef01d4c50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 >> v1:172.21.15.146:6789/0 conn(0x7fcef006e960 legacy=0x7fcef01cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: _finish_hunting 0 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 monclient: found mon.a 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:32:09.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef01274b0 con 0x7fcef009f5a0 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: _finish_auth 0 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: _check_auth_tickets 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 5 monclient: authenticate success, global_id 4458 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient: _renew_subs 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcef01262e0 con 0x7fcef009f5a0 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 client.4458.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient: _renew_subs 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcef01344f0 con 0x7fcef009f5a0 2024-09-15T20:32:09.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 librados: init done 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcee4004350 con 0x7fcef009f5a0 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fceed7fa640 4 set_mon_vals no callback set 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcee40047e0 con 0x7fcef009f5a0 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: dump: 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.329 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fced00065d0 con 0x7fced0002870 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 client.4458.objecter ms_handle_connect 0x7fced0002870 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 client.4458.objecter resend_mon_ops 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fcee4004a00 con 0x7fcef009f5a0 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7fcee404b120 con 0x7fcef009f5a0 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 client.4458.objecter ms_dispatch 0x7fcef0000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 3 client.4458.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 3 client.4458.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 20 client.4458.objecter dump_active .. 0 homeless 2024-09-15T20:32:09.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482372645 0 0) 0x7fced00065d0 con 0x7fced0002870 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient(hunting): my global_id is 4461 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcef0128360 con 0x7fced0002870 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2196641202 0 0) 0x7fcef0128360 con 0x7fced0002870 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fced00065d0 con 0x7fced0002870 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcee0001720 con 0x7fced0002870 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: dump: 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.331 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fceeffff640 10 monclient: get_auth_request con 0x7fcedc03e0d0 auth_method 0 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2984761687 0 0) 0x7fced00065d0 con 0x7fced0002870 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fced000a170 con 0x7fced0002870 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 client.4458.objecter ms_handle_connect 0x7fcedc03e0d0 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcee00019e0 con 0x7fced0002870 2024-09-15T20:32:09.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:32:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:32:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 >> v1:172.21.15.78:6789/0 conn(0x7fced0002870 legacy=0x7fced0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.311+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 >> v1:172.21.15.78:6789/0 conn(0x7fced0002870 legacy=0x7fced0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:32:09.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:32:09.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fced0009e80 con 0x7fced0009140 2024-09-15T20:32:09.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 client.4458.objecter ms_handle_connect 0x7fced0009140 2024-09-15T20:32:09.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 client.4458.objecter resend_mon_ops 2024-09-15T20:32:09.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 794906917 0 0) 0x7fced0009e80 con 0x7fced0009140 2024-09-15T20:32:09.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient(hunting): my global_id is 4464 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fced00065d0 con 0x7fced0009140 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1647829073 0 0) 0x7fced00065d0 con 0x7fced0009140 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fced0009e80 con 0x7fced0009140 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcee00037a0 con 0x7fced0009140 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient: dump: 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.335 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2068021186 0 0) 0x7fced0009e80 con 0x7fced0009140 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fced000c420 con 0x7fced0009140 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4127 (unknown 0 0 3565297252) 0x7fcee0003730 con 0x7fced0009140 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 >> v1:172.21.15.78:6789/0 conn(0x7fced0009140 legacy=0x7fced00095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fced7fff640 1 -- 172.21.15.78:0/4153750773 >> v1:172.21.15.78:6789/0 conn(0x7fced0009140 legacy=0x7fced00095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 8, 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 41871, 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.337 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:32:09.338 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:32:09.339 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:32:09.340 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:32:09.341 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:32:09.342 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:32:09.343 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.344 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:32:09.345 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.346 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:32:09.347 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:32:09.348 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:32:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 10 librados: watch_flush enter 2024-09-15T20:32:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 10 librados: watch_flush exit 2024-09-15T20:32:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 20 client.4458.objecter shutdown clearing up homeless session... 2024-09-15T20:32:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 10 client.4458.objecter successfully canceled tick 2024-09-15T20:32:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 >> v1:172.21.15.146:6812/435598642 conn(0x7fcedc03e0d0 legacy=0x7fcedc040590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 10 monclient: shutdown 2024-09-15T20:32:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:32:09.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 >> v1:172.21.15.78:6789/0 conn(0x7fcef009f5a0 legacy=0x7fcef0133c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fceeffff640 1 -- 172.21.15.78:0/4153750773 reap_dead start 2024-09-15T20:32:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fceeffff640 1 -- 172.21.15.78:0/4153750773 reap_dead start 2024-09-15T20:32:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 shutdown_connections 2024-09-15T20:32:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 >> 172.21.15.78:0/4153750773 conn(0x7fcef00a63c0 msgr2=0x7fcef009d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:32:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 shutdown_connections 2024-09-15T20:32:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 1 -- 172.21.15.78:0/4153750773 wait complete. 2024-09-15T20:32:09.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.319+0000 7fcef5da8640 1 librados: shutdown 2024-09-15T20:32:09.351 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:32:09.351 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T20:32:09.351 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:32:09.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: get_monmap_and_config 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: build_initial_monmap 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: monmap: 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:32:09.465144+0000 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:32:09.465144+0000 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:32:09.466 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 Processor -- start 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- start start 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: init 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: _reopen_session rank -1 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient(hunting): picked mon.noname-a con 0x7f817c09ca50 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient(hunting): picked mon.noname-c con 0x7f817c09d540 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient(hunting): picked mon.noname-b con 0x7f817c0a56f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:09.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c0a6c50 con 0x7f817c09ca50 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c0a4d50 con 0x7f817c09d540 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c0a8ca0 con 0x7f817c0a56f0 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient(hunting): _renew_subs 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient(hunting): authenticate will time out at 1190.349757s 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817bfff640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f817c0a56f0 legacy=0x7f817c140690 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f817c09ca50 0x7f817c09ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34746/0 (socket says 172.21.15.78:34746) 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817b7fe640 1 -- 172.21.15.78:0/1478404845 learned_addr learned my addr 172.21.15.78:0/1478404845 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1619052637 0 0) 0x7f817c0a6c50 con 0x7f817c09ca50 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient(hunting): my global_id is 4467 2024-09-15T20:32:09.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8168003530 con 0x7f817c09ca50 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2853587196 0 0) 0x7f8168003530 con 0x7f817c09ca50 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 >> v1:172.21.15.78:6790/0 conn(0x7f817c0a56f0 legacy=0x7f817c140690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 >> v1:172.21.15.146:6789/0 conn(0x7f817c09d540 legacy=0x7f817c13cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 monclient: found mon.noname-a 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c0a5b40 con 0x7f817c09ca50 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: _finish_auth 0 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:32:09.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f81740016a0 con 0x7f817c09ca50 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: dump: 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.477 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 5 monclient: authenticate success, global_id 4467 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f81740018c0 con 0x7f817c09ca50 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 1 -- 172.21.15.78:0/1478404845 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8174001d50 con 0x7f817c09ca50 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: get_monmap_and_config success 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 4 set_mon_vals no callback set 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f817a7fc640 10 monclient: dump: 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.478 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: shutdown 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- 172.21.15.78:0/1478404845 >> v1:172.21.15.78:6789/0 conn(0x7f817c09ca50 legacy=0x7f817c09ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- 172.21.15.78:0/1478404845 shutdown_connections 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- 172.21.15.78:0/1478404845 >> 172.21.15.78:0/1478404845 conn(0x7f817c0aa520 msgr2=0x7f817c0aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- 172.21.15.78:0/1478404845 shutdown_connections 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 -- 172.21.15.78:0/1478404845 wait complete. 2024-09-15T20:32:09.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: build_initial_monmap 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 10 monclient: monmap: 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 librados: starting msgr at 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 librados: starting objecter 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.463+0000 7f8181e12640 1 Processor -- start 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 -- start start 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 librados: setting wanted keys 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 librados: calling monclient init 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient: init 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient: _reopen_session rank -1 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient(hunting): picked mon.noname-c con 0x7f817c09ca50 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient(hunting): picked mon.noname-b con 0x7f817c09d540 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient(hunting): picked mon.noname-a con 0x7f817c0a56f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:09.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c13f610 con 0x7f817c0a56f0 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f817b7fe640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f817c09ca50 legacy=0x7f817c13c660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c13fbe0 con 0x7f817c09d540 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f817c13d480 con 0x7f817c09ca50 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient(hunting): _renew_subs 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient(hunting): authenticate will time out at 1190.352625s 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f817bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f817c0a56f0 0x7f817c1d4f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34750/0 (socket says 172.21.15.78:34750) 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f817bfff640 1 -- 172.21.15.78:0/3189695773 learned_addr learned my addr 172.21.15.78:0/3189695773 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 client.?.objecter ms_handle_connect 0x7f817c0a56f0 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4017142584 0 0) 0x7f817c13f610 con 0x7f817c0a56f0 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): my global_id is 4470 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f815c004ec0 con 0x7f817c0a56f0 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 client.?.objecter ms_handle_connect 0x7f817c09d540 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3633245712 0 0) 0x7f815c004ec0 con 0x7f817c0a56f0 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f817c13f610 con 0x7f817c0a56f0 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f816c0040c0 con 0x7f817c0a56f0 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): dump: 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.483 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 514583717 0 0) 0x7f817c13f610 con 0x7f817c0a56f0 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 >> v1:172.21.15.78:6790/0 conn(0x7f817c09ca50 legacy=0x7f817c13c660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 >> v1:172.21.15.146:6789/0 conn(0x7f817c09d540 legacy=0x7f817c13cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: _finish_hunting 0 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 monclient: found mon.a 2024-09-15T20:32:09.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f817c13e120 con 0x7f817c0a56f0 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: _finish_auth 0 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: _check_auth_tickets 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 5 monclient: authenticate success, global_id 4470 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient: _renew_subs 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f817c140480 con 0x7f817c0a56f0 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 client.4470.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient: _renew_subs 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:32:09.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f817c138a00 con 0x7f817c0a56f0 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 librados: init done 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f816c0042e0 con 0x7f817c0a56f0 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8179ffb640 4 set_mon_vals no callback set 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f816c0046f0 con 0x7f817c0a56f0 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: dump: 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.486 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f81600065d0 con 0x7f8160002870 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f816c004910 con 0x7f817c0a56f0 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7f816c04b0a0 con 0x7f817c0a56f0 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 client.4470.objecter ms_dispatch 0x7f817c000ba0 osd_map(20..20 src has 1..20) 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 3 client.4470.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 3 client.4470.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 20 client.4470.objecter dump_active .. 0 homeless 2024-09-15T20:32:09.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f817affd640 10 monclient: get_auth_request con 0x7f815c03de90 auth_method 0 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 client.4470.objecter ms_handle_connect 0x7f8160002870 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 client.4470.objecter resend_mon_ops 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1813932617 0 0) 0x7f81600065d0 con 0x7f8160002870 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient(hunting): my global_id is 4456 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f817c13f610 con 0x7f8160002870 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 client.4470.objecter ms_handle_connect 0x7f815c03de90 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4206523445 0 0) 0x7f817c13f610 con 0x7f8160002870 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f81600065d0 con 0x7f8160002870 2024-09-15T20:32:09.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8174001ce0 con 0x7f8160002870 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 10 monclient: dump: 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.489 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1772240105 0 0) 0x7f81600065d0 con 0x7f8160002870 2024-09-15T20:32:09.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.467+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f816000a170 con 0x7f8160002870 2024-09-15T20:32:09.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.471+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8174001ce0 con 0x7f8160002870 2024-09-15T20:32:09.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.471+0000 7f8173fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:32:09.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.471+0000 7f8173fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:32:09.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.471+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 >> v1:172.21.15.146:6789/0 conn(0x7f8160002870 legacy=0x7f8160004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.471+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 >> v1:172.21.15.146:6789/0 conn(0x7f8160002870 legacy=0x7f8160004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8181e12640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8160009e80 con 0x7f8160009140 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8173fff640 10 client.4470.objecter ms_handle_connect 0x7f8160009140 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8173fff640 10 client.4470.objecter resend_mon_ops 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985189498 0 0) 0x7f8160009e80 con 0x7f8160009140 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8173fff640 10 monclient(hunting): _negotiate 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8173fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8173fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8173fff640 10 monclient(hunting): my global_id is 4459 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.475+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f81600065d0 con 0x7f8160009140 2024-09-15T20:32:09.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 923137571 0 0) 0x7f81600065d0 con 0x7f8160009140 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8160009e80 con 0x7f8160009140 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f81740049d0 con 0x7f8160009140 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 10 monclient: dump: 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:09.493 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2939354049 0 0) 0x7f8160009e80 con 0x7f8160009140 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f816000c420 con 0x7f8160009140 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4292 (unknown 0 0 383959235) 0x7f81740049d0 con 0x7f8160009140 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 >> v1:172.21.15.146:6789/0 conn(0x7f8160009140 legacy=0x7f81600095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8173fff640 1 -- 172.21.15.78:0/3189695773 >> v1:172.21.15.146:6789/0 conn(0x7f8160009140 legacy=0x7f81600095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:32:09.494 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 8, 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 41987, 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:32:09.495 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:32:09.496 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.497 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:32:09.498 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:32:09.499 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:32:09.500 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:32:09.501 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:32:09.502 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.503 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:32:09.504 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:32:09.505 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:32:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:32:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:32:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:32:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:32:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:32:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:32:09.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:32:09.507 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:32:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:32:09.508 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:32:09.508 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:32:09.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 10 librados: watch_flush enter 2024-09-15T20:32:09.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 10 librados: watch_flush exit 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 20 client.4470.objecter shutdown clearing up homeless session... 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 10 client.4470.objecter successfully canceled tick 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 >> v1:172.21.15.146:6812/435598642 conn(0x7f815c03de90 legacy=0x7f815c040350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 10 monclient: shutdown 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 >> v1:172.21.15.78:6789/0 conn(0x7f817c0a56f0 legacy=0x7f817c1d4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f817bfff640 1 -- 172.21.15.78:0/3189695773 reap_dead start 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 shutdown_connections 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 >> 172.21.15.78:0/3189695773 conn(0x7f817c0aa520 msgr2=0x7f817c0a4160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 shutdown_connections 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 1 -- 172.21.15.78:0/3189695773 wait complete. 2024-09-15T20:32:09.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:32:09.479+0000 7f8181e12640 1 librados: shutdown 2024-09-15T20:32:09.511 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:32:09.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:32:47.279 INFO:tasks.workunit.client.0.smithi146.stderr:Note: switching to '7498493279982dfe87f00616198a5967475f6169'. 2024-09-15T20:32:47.279 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-15T20:32:47.279 INFO:tasks.workunit.client.0.smithi146.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr:state without impacting any branches by switching back to a branch. 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr: git switch -c 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr:Or undo this operation with: 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr: git switch - 2024-09-15T20:32:47.280 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-15T20:32:47.281 INFO:tasks.workunit.client.0.smithi146.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-15T20:32:47.281 INFO:tasks.workunit.client.0.smithi146.stderr: 2024-09-15T20:32:47.281 INFO:tasks.workunit.client.0.smithi146.stderr:HEAD is now at 74984932799 Merge pull request #59797 from zdover23/wip-doc-2024-09-15-install-install-storage-cluster 2024-09-15T20:32:47.282 DEBUG:teuthology.orchestra.run.smithi146:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-09-15T20:32:47.330 INFO:tasks.workunit.client.0.smithi146.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-15T20:32:47.333 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T20:32:47.333 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-15T20:32:48.246 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-15T20:32:48.346 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-15T20:32:48.398 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-09-15T20:32:48.401 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T20:32:48.401 INFO:tasks.workunit.client.0.smithi146.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-15T20:32:48.453 INFO:tasks.workunit.client.0.smithi146.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-09-15T20:32:48.456 DEBUG:teuthology.orchestra.run.smithi146:> set -ex 2024-09-15T20:32:48.456 DEBUG:teuthology.orchestra.run.smithi146:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-09-15T20:32:48.506 INFO:tasks.workunit:Running workunits matching rados/test.sh on client.0... 2024-09-15T20:32:48.507 INFO:tasks.workunit:Running workunit rados/test.sh... 2024-09-15T20:32:48.507 DEBUG:teuthology.orchestra.run.smithi146:workunit test rados/test.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7498493279982dfe87f00616198a5967475f6169 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/rados/test.sh 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ parallel=1 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' '' = --serial ']' 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ crimson=0 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' '' = --crimson ']' 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ color= 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -t 1 ']' 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ trap cleanup EXIT ERR HUP INT QUIT 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ GTEST_OUTPUT_DIR=/home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-15T20:32:48.560 INFO:tasks.workunit.client.0.smithi146.stderr:+ mkdir -p /home/ubuntu/cephtest/archive/unit_test_xml_report 2024-09-15T20:32:48.561 INFO:tasks.workunit.client.0.smithi146.stderr:+ declare -A pids 2024-09-15T20:32:48.561 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.561 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.561 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_aio 2024-09-15T20:32:48.562 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_aio' 2024-09-15T20:32:48.562 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_aio 2024-09-15T20:32:48.562 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.564 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_aio 2024-09-15T20:32:48.565 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28164 2024-09-15T20:32:48.565 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_aio on pid 28164' 2024-09-15T20:32:48.565 INFO:tasks.workunit.client.0.smithi146.stdout:test api_aio on pid 28164 2024-09-15T20:32:48.566 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28164 2024-09-15T20:32:48.566 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.566 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2>&1 | tee ceph_test_rados_api_aio.log | sed "s/^/ api_aio: /"' 2024-09-15T20:32:48.566 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.566 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_aio_pp 2024-09-15T20:32:48.566 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_aio_pp' 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_aio_pp 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_aio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio.xml 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_aio.log 2024-09-15T20:32:48.567 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_aio: /' 2024-09-15T20:32:48.568 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_aio_pp 2024-09-15T20:32:48.568 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28172 2024-09-15T20:32:48.568 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_aio_pp on pid 28172' 2024-09-15T20:32:48.568 INFO:tasks.workunit.client.0.smithi146.stdout:test api_aio_pp on pid 28172 2024-09-15T20:32:48.569 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28172 2024-09-15T20:32:48.569 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2>&1 | tee ceph_test_rados_api_aio_pp.log | sed "s/^/ api_aio_pp: /"' 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_io 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_io' 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_io 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.570 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_aio_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_aio_pp.xml 2024-09-15T20:32:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_aio_pp.log 2024-09-15T20:32:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_aio_pp: /' 2024-09-15T20:32:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_io 2024-09-15T20:32:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28180 2024-09-15T20:32:48.571 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_io on pid 28180' 2024-09-15T20:32:48.572 INFO:tasks.workunit.client.0.smithi146.stdout:test api_io on pid 28180 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28180 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2>&1 | tee ceph_test_rados_api_io.log | sed "s/^/ api_io: /"' 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_io_pp 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_io_pp' 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_io_pp 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.573 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.574 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.574 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.574 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_io --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io.xml 2024-09-15T20:32:48.574 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_io.log 2024-09-15T20:32:48.574 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_io: /' 2024-09-15T20:32:48.574 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_io_pp 2024-09-15T20:32:48.575 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28188 2024-09-15T20:32:48.575 INFO:tasks.workunit.client.0.smithi146.stdout:test api_io_pp on pid 28188 2024-09-15T20:32:48.576 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_io_pp on pid 28188' 2024-09-15T20:32:48.576 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28188 2024-09-15T20:32:48.576 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2>&1 | tee ceph_test_rados_api_io_pp.log | sed "s/^/ api_io_pp: /"' 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_asio 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_asio' 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_asio 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_io_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_io_pp.xml 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_io_pp.log 2024-09-15T20:32:48.577 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_io_pp: /' 2024-09-15T20:32:48.578 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_asio 2024-09-15T20:32:48.579 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28196 2024-09-15T20:32:48.579 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_asio on pid 28196' 2024-09-15T20:32:48.579 INFO:tasks.workunit.client.0.smithi146.stdout:test api_asio on pid 28196 2024-09-15T20:32:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28196 2024-09-15T20:32:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2>&1 | tee ceph_test_rados_api_asio.log | sed "s/^/ api_asio: /"' 2024-09-15T20:32:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_list 2024-09-15T20:32:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_list' 2024-09-15T20:32:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_list 2024-09-15T20:32:48.580 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.581 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.581 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.581 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.581 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.582 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_asio --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_asio.xml 2024-09-15T20:32:48.582 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_asio.log 2024-09-15T20:32:48.582 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_asio: /' 2024-09-15T20:32:48.582 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_list 2024-09-15T20:32:48.583 INFO:tasks.workunit.client.0.smithi146.stdout:test api_list on pid 28204 2024-09-15T20:32:48.583 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28204 2024-09-15T20:32:48.583 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_list on pid 28204' 2024-09-15T20:32:48.584 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28204 2024-09-15T20:32:48.584 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.584 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2>&1 | tee ceph_test_rados_api_list.log | sed "s/^/ api_list: /"' 2024-09-15T20:32:48.584 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.584 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_lock 2024-09-15T20:32:48.584 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_lock' 2024-09-15T20:32:48.585 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_lock 2024-09-15T20:32:48.585 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.585 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.585 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.585 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.585 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.585 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_list --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_list.xml 2024-09-15T20:32:48.585 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_list: /' 2024-09-15T20:32:48.586 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_list.log 2024-09-15T20:32:48.587 INFO:tasks.workunit.client.0.smithi146.stdout:test api_lock on pid 28212 2024-09-15T20:32:48.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_lock 2024-09-15T20:32:48.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28212 2024-09-15T20:32:48.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_lock on pid 28212' 2024-09-15T20:32:48.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28212 2024-09-15T20:32:48.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.588 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_lock_pp 2024-09-15T20:32:48.588 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_lock_pp' 2024-09-15T20:32:48.589 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2>&1 | tee ceph_test_rados_api_lock.log | sed "s/^/ api_lock: /"' 2024-09-15T20:32:48.589 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_lock_pp 2024-09-15T20:32:48.589 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.590 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.590 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.590 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.590 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.591 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_lock: /' 2024-09-15T20:32:48.592 INFO:tasks.workunit.client.0.smithi146.stdout:test api_lock_pp on pid 28220 2024-09-15T20:32:48.592 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_lock_pp 2024-09-15T20:32:48.592 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28220 2024-09-15T20:32:48.592 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_lock_pp on pid 28220' 2024-09-15T20:32:48.592 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28220 2024-09-15T20:32:48.592 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_lock --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock.xml 2024-09-15T20:32:48.593 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.593 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.593 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_lock.log 2024-09-15T20:32:48.593 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2>&1 | tee ceph_test_rados_api_lock_pp.log | sed "s/^/ api_lock_pp: /"' 2024-09-15T20:32:48.593 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_misc 2024-09-15T20:32:48.593 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_misc' 2024-09-15T20:32:48.594 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_misc 2024-09-15T20:32:48.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.596 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_lock_pp: /' 2024-09-15T20:32:48.597 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_lock_pp.log 2024-09-15T20:32:48.598 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.601 INFO:tasks.workunit.client.0.smithi146.stdout:test api_misc on pid 28228 2024-09-15T20:32:48.601 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_misc 2024-09-15T20:32:48.602 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28228 2024-09-15T20:32:48.602 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_misc on pid 28228' 2024-09-15T20:32:48.602 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28228 2024-09-15T20:32:48.602 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.602 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.602 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2>&1 | tee ceph_test_rados_api_misc.log | sed "s/^/ api_misc: /"' 2024-09-15T20:32:48.603 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.603 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.603 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.603 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.603 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_lock_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_lock_pp.xml 2024-09-15T20:32:48.604 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_misc_pp 2024-09-15T20:32:48.604 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_misc_pp' 2024-09-15T20:32:48.610 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_misc --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc.xml 2024-09-15T20:32:48.615 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_misc_pp 2024-09-15T20:32:48.615 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.616 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_misc.log 2024-09-15T20:32:48.616 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_misc: /' 2024-09-15T20:32:48.618 INFO:tasks.workunit.client.0.smithi146.stdout:test api_misc_pp on pid 28266 2024-09-15T20:32:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_misc_pp 2024-09-15T20:32:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28266 2024-09-15T20:32:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_misc_pp on pid 28266' 2024-09-15T20:32:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28266 2024-09-15T20:32:48.619 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.620 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.620 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2>&1 | tee ceph_test_rados_api_misc_pp.log | sed "s/^/ api_misc_pp: /"' 2024-09-15T20:32:48.620 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_tier_pp 2024-09-15T20:32:48.620 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_tier_pp' 2024-09-15T20:32:48.620 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.621 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_misc_pp.log 2024-09-15T20:32:48.622 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_misc_pp: /' 2024-09-15T20:32:48.623 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_tier_pp 2024-09-15T20:32:48.624 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.627 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_misc_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_misc_pp.xml 2024-09-15T20:32:48.630 INFO:tasks.workunit.client.0.smithi146.stdout:test api_tier_pp on pid 28316 2024-09-15T20:32:48.630 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_tier_pp 2024-09-15T20:32:48.630 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28316 2024-09-15T20:32:48.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_tier_pp on pid 28316' 2024-09-15T20:32:48.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28316 2024-09-15T20:32:48.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.631 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2>&1 | tee ceph_test_rados_api_tier_pp.log | sed "s/^/ api_tier_pp: /"' 2024-09-15T20:32:48.632 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.632 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.632 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.632 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.632 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_tier_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_tier_pp.xml 2024-09-15T20:32:48.635 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_pool 2024-09-15T20:32:48.635 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_pool' 2024-09-15T20:32:48.635 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_tier_pp.log 2024-09-15T20:32:48.636 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.637 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_pool 2024-09-15T20:32:48.637 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_tier_pp: /' 2024-09-15T20:32:48.638 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_pool 2024-09-15T20:32:48.638 INFO:tasks.workunit.client.0.smithi146.stdout:test api_pool on pid 28333 2024-09-15T20:32:48.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28333 2024-09-15T20:32:48.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_pool on pid 28333' 2024-09-15T20:32:48.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28333 2024-09-15T20:32:48.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.639 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2>&1 | tee ceph_test_rados_api_pool.log | sed "s/^/ api_pool: /"' 2024-09-15T20:32:48.639 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_snapshots 2024-09-15T20:32:48.640 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_snapshots' 2024-09-15T20:32:48.640 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.640 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.640 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.640 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.640 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_pool --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_pool.xml 2024-09-15T20:32:48.641 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_pool.log 2024-09-15T20:32:48.642 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.646 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_snapshots 2024-09-15T20:32:48.648 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_snapshots 2024-09-15T20:32:48.650 INFO:tasks.workunit.client.0.smithi146.stdout:test api_snapshots on pid 28358 2024-09-15T20:32:48.650 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [==========] Running 12 tests from 1 test suite. 2024-09-15T20:32:48.650 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [----------] Global test environment set-up. 2024-09-15T20:32:48.650 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [----------] 12 tests from AsioRados 2024-09-15T20:32:48.650 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient: get_monmap_and_config 2024-09-15T20:32:48.650 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient: build_initial_monmap 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient: monmap: 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: epoch 0 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: last_changed 0.000000 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: created 0.000000 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: election_strategy: 1 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 1 Processor -- start 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 1 -- start start 2024-09-15T20:32:48.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient: init 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient: _reopen_session rank -1 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient(hunting): picked mon.noname-b con 0x561c1df24c00 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient(hunting): picked mon.noname-c con 0x561c1df24fe0 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient(hunting): picked mon.noname-a con 0x561c1df4cf60 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x561c1e0779e0 con 0x561c1df4cf60 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x561c1df1fe80 con 0x561c1df24c00 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x561c1df234c0 con 0x561c1df24fe0 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient(hunting): _renew_subs 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed3a46880 10 monclient(hunting): authenticate will time out at 1218.554777s 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed1550640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x561c1df24c00 0x561c1e076bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.146:35796/0 (socket says 172.21.15.146:35796) 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed0d4f640 1 -- >> v1:172.21.15.78:6790/0 conn(0x561c1df24fe0 legacy=0x561c1e0772d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.641+0000 7faed1550640 1 -- 172.21.15.146:0/2663567321 learned_addr learned my addr 172.21.15.146:0/2663567321 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 277977185 0 0) 0x561c1e0779e0 con 0x561c1df4cf60 2024-09-15T20:32:48.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:32:48.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:48.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28358 2024-09-15T20:32:48.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_snapshots on pid 28358' 2024-09-15T20:32:48.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28358 2024-09-15T20:32:48.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.653 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_snapshots_pp 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_snapshots_pp' 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2>&1 | tee ceph_test_rados_api_snapshots.log | sed "s/^/ api_snapshots: /"' 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_pool: /' 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient(hunting): my global_id is 4494 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faeb4001e20 con 0x561c1df4cf60 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3956197510 0 0) 0x7faeb4001e20 con 0x561c1df4cf60 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 >> v1:172.21.15.78:6790/0 conn(0x561c1df24fe0 legacy=0x561c1e0772d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 >> v1:172.21.15.146:6789/0 conn(0x561c1df24c00 legacy=0x561c1e076bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:48.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: _finish_hunting 0 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 monclient: found mon.noname-a 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c1df4c520 con 0x561c1df4cf60 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: _finish_auth 0 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: _check_auth_tickets 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 5 monclient: authenticate success, global_id 4494 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faec4001bc0 con 0x561c1df4cf60 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:32:48.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: dump: 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: epoch 1 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: min_mon_release 19 (squid) 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: election_strategy: 1 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faec4001370 con 0x561c1df4cf60 2024-09-15T20:32:48.656 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:32:48.657 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 1 -- 172.21.15.146:0/2663567321 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faec40030e0 con 0x561c1df4cf60 2024-09-15T20:32:48.657 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.657 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.657 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.657 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:48.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:32:48.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faec37fe640 10 monclient: dump: 2024-09-15T20:32:48.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: epoch 1 2024-09-15T20:32:48.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:48.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: min_mon_release 19 (squid) 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: election_strategy: 1 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 10 monclient: get_monmap_and_config success 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 4 set_mon_vals no callback set 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 10 monclient: shutdown 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:32:48.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 1 -- 172.21.15.146:0/2663567321 >> v1:172.21.15.78:6789/0 conn(0x561c1df4cf60 legacy=0x561c1e07f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 1 -- 172.21.15.146:0/2663567321 shutdown_connections 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 1 -- 172.21.15.146:0/2663567321 >> 172.21.15.146:0/2663567321 conn(0x561c1df1ce60 msgr2=0x561c1df24100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 1 -- 172.21.15.146:0/2663567321 shutdown_connections 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 1 -- 172.21.15.146:0/2663567321 wait complete. 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 10 monclient: build_initial_monmap 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.645+0000 7faed3a46880 10 monclient: monmap: 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: epoch 0 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: last_changed 0.000000 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: created 0.000000 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: min_mon_release 0 (unknown) 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: election_strategy: 1 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:32:48.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 librados: starting msgr at 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 librados: starting objecter 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 Processor -- start 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 -- start start 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 librados: setting wanted keys 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 librados: calling monclient init 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 10 monclient: init 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 10 monclient: _reopen_session rank -1 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 10 monclient(hunting): picked mon.noname-b con 0x561c1df24c00 addr v1:172.21.15.146:6789/0 2024-09-15T20:32:48.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 10 monclient(hunting): picked mon.noname-c con 0x561c1df24fe0 addr v1:172.21.15.78:6790/0 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 10 monclient(hunting): picked mon.noname-a con 0x561c1df4cf60 addr v1:172.21.15.78:6789/0 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x561c1e07ed40 con 0x561c1df4cf60 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x561c1e0570f0 con 0x561c1df24c00 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x561c1e057390 con 0x561c1df24fe0 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 10 monclient(hunting): _renew_subs 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed3a46880 10 monclient(hunting): authenticate will time out at 1218.561225s 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed1d51640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x561c1df4cf60 0x561c1e086180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.146:60494/0 (socket says 172.21.15.146:60494) 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 learned_addr learned my addr 172.21.15.146:0/1037512150 (peer_addr_for_me v1:172.21.15.146:0/0) 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faed0d4f640 1 -- 172.21.15.146:0/1037512150 >> v1:172.21.15.78:6790/0 conn(0x561c1df24fe0 legacy=0x561c1e07e630 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 client.?.objecter ms_handle_connect 0x561c1df4cf60 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 client.?.objecter ms_handle_connect 0x561c1df24c00 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:32:48.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1942147277 0 0) 0x561c1e07ed40 con 0x561c1df4cf60 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): my global_id is 4506 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faeb4001d20 con 0x561c1df4cf60 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054827812 0 0) 0x561c1e0570f0 con 0x561c1df24c00 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): my global_id is 4504 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x561c1e07ed40 con 0x561c1df24c00 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 481058886 0 0) 0x7faeb4001d20 con 0x561c1df4cf60 2024-09-15T20:32:48.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x561c1e0570f0 con 0x561c1df4cf60 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2377350345 0 0) 0x561c1e07ed40 con 0x561c1df24c00 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faeb4001d20 con 0x561c1df24c00 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faec4001490 con 0x561c1df4cf60 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): dump: 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: epoch 1 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: min_mon_release 19 (squid) 2024-09-15T20:32:48.664 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: election_strategy: 1 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faec8001900 con 0x561c1df24c00 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.649+0000 7faec27fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient(hunting): dump: 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: epoch 1 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.665 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: min_mon_release 19 (squid) 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: election_strategy: 1 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1822210724 0 0) 0x7faeb4001d20 con 0x561c1df24c00 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 >> v1:172.21.15.78:6790/0 conn(0x561c1df24fe0 legacy=0x561c1e07e630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 >> v1:172.21.15.78:6789/0 conn(0x561c1df4cf60 legacy=0x561c1e086180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient: _finish_hunting 0 2024-09-15T20:32:48.666 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 1 monclient: found mon.b 2024-09-15T20:32:48.667 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_snapshots --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots.xml 2024-09-15T20:32:48.667 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_snapshots: /' 2024-09-15T20:32:48.667 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_snapshots.log 2024-09-15T20:32:48.667 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:32:48.667 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:32:48.667 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x561c1e057630 con 0x561c1df24c00 2024-09-15T20:32:48.667 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient: _finish_auth 0 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient: _check_auth_tickets 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 5 monclient: authenticate success, global_id 4504 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 10 monclient: _renew_subs 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x561c1e0768f0 con 0x561c1df24c00 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 10 monclient: _renew_subs 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x561c1df1ef40 con 0x561c1df24c00 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 1 librados: init done 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed3a46880 10 librados: wait_for_osdmap waiting 2024-09-15T20:32:48.668 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faec8001e30 con 0x561c1df24c00 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faec8002240 con 0x561c1df24c00 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 10 monclient: dump: 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: epoch 1 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: min_mon_release 19 (squid) 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: election_strategy: 1 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:32:48.669 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:32:48.670 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:32:48.670 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.670 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7faec80023e0 con 0x561c1df24c00 2024-09-15T20:32:48.670 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.653+0000 7faed0d4f640 10 monclient: get_auth_request con 0x7faeb403c430 auth_method 0 2024-09-15T20:32:48.670 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec27fc640 10 client.4504.objecter ms_handle_connect 0x7faeb403c430 2024-09-15T20:32:48.670 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_snapshots_pp 2024-09-15T20:32:48.670 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.671 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:67test api_snapshots_pp on pid 28407 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_snapshots_pp 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28407 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_snapshots_pp on pid 28407' 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28407 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2>&1 | tee ceph_test_rados_api_snapshots_pp.log | sed "s/^/ api_snapshots_pp: /"' 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_stat 2024-09-15T20:32:48.672 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_stat' 2024-09-15T20:32:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_snapshots_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_snapshots_pp.xml 2024-09-15T20:32:48.673 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_stat 2024-09-15T20:32:48.673 INFO:tasks.workunit.client.0.smithi146.stdout:test api_stat on pid 28416 2024-09-15T20:32:48.674 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_stat 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28416 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_stat on pid 28416' 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28416 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2>&1 | tee ceph_test_rados_api_stat.log | sed "s/^/ api_stat: /"' 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_stat_pp 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_stat_pp' 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_stat_pp 2024-09-15T20:32:48.675 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.676 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.676 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.676 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_stat.log 2024-09-15T20:32:48.676 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_stat --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat.xml 2024-09-15T20:32:48.676 INFO:tasks.workunit.client.0.smithi146.stdout:test api_stat_pp on pid 28427 2024-09-15T20:32:48.676 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_stat_pp 2024-09-15T20:32:48.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28427 2024-09-15T20:32:48.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_stat_pp on pid 28427' 2024-09-15T20:32:48.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28427 2024-09-15T20:32:48.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.677 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2>&1 | tee ceph_test_rados_api_stat_pp.log | sed "s/^/ api_stat_pp: /"' 2024-09-15T20:32:48.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_snapshots_pp.log 2024-09-15T20:32:48.678 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_watch_notify 2024-09-15T20:32:48.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_watch_notify' 2024-09-15T20:32:48.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_stat: /' 2024-09-15T20:32:48.678 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_snapshots_pp: /' 2024-09-15T20:32:48.679 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_stat_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_stat_pp.xml 2024-09-15T20:32:48.680 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_watch_notify 2024-09-15T20:32:48.681 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.682 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_stat_pp.log 2024-09-15T20:32:48.682 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_stat_pp: /' 2024-09-15T20:32:48.687 INFO:tasks.workunit.client.0.smithi146.stdout:test api_watch_notify on pid 28457 2024-09-15T20:32:48.687 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_watch_notify 2024-09-15T20:32:48.687 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28457 2024-09-15T20:32:48.687 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_watch_notify on pid 28457' 2024-09-15T20:32:48.687 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28457 2024-09-15T20:32:48.688 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.688 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.688 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2>&1 | tee ceph_test_rados_api_watch_notify.log | sed "s/^/ api_watch_notify: /"' 2024-09-15T20:32:48.688 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_watch_notify_pp 2024-09-15T20:32:48.688 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_watch_notify_pp' 2024-09-15T20:32:48.691 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.691 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.691 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.691 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.692 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_watch_notify: /' 2024-09-15T20:32:48.694 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_watch_notify --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify.xml 2024-09-15T20:32:48.696 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_watch_notify.log 2024-09-15T20:32:48.699 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_watch_notify_pp 2024-09-15T20:32:48.699 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.702 INFO:tasks.workunit.client.0.smithi146.stdout:test api_watch_notify_pp on pid 28495 2024-09-15T20:32:48.702 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_watch_notify_pp 2024-09-15T20:32:48.702 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28495 2024-09-15T20:32:48.702 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_watch_notify_pp on pid 28495' 2024-09-15T20:32:48.702 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28495 2024-09-15T20:32:48.702 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.702 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.703 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2>&1 | tee ceph_test_rados_api_watch_notify_pp.log | sed "s/^/ api_watch_notify_pp: /"' 2024-09-15T20:32:48.704 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.705 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.705 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.705 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.705 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_watch_notify_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_watch_notify_pp.xml 2024-09-15T20:32:48.709 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_cmd 2024-09-15T20:32:48.709 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_watch_notify_pp: /' 2024-09-15T20:32:48.709 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_cmd' 2024-09-15T20:32:48.709 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_watch_notify_pp.log 2024-09-15T20:32:48.713 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_cmd 2024-09-15T20:32:48.723 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.726 INFO:tasks.workunit.client.0.smithi146.stdout:test api_cmd on pid 28577 2024-09-15T20:32:48.726 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_cmd 2024-09-15T20:32:48.726 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28577 2024-09-15T20:32:48.726 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_cmd on pid 28577' 2024-09-15T20:32:48.726 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28577 2024-09-15T20:32:48.726 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.726 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.726 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2>&1 | tee ceph_test_rados_api_cmd.log | sed "s/^/ api_cmd: /"' 2024-09-15T20:32:48.727 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.727 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.727 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.728 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.728 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_cmd --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd.xml 2024-09-15T20:32:48.728 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_cmd.log 2024-09-15T20:32:48.728 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_cmd: /' 2024-09-15T20:32:48.729 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_cmd_pp 2024-09-15T20:32:48.729 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_cmd_pp' 2024-09-15T20:32:48.729 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_cmd_pp 2024-09-15T20:32:48.732 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.734 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_cmd_pp 2024-09-15T20:32:48.734 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28599 2024-09-15T20:32:48.734 INFO:tasks.workunit.client.0.smithi146.stdout:test api_cmd_pp on pid 28599 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_cmd_pp on pid 28599' 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28599 2024-09-15T20:32:48.735 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2>&1 | tee ceph_test_rados_api_cmd_pp.log | sed "s/^/ api_cmd_pp: /"' 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_service 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_service' 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_service 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_cmd_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_cmd_pp.xml 2024-09-15T20:32:48.736 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_cmd_pp.log 2024-09-15T20:32:48.737 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_cmd_pp: /' 2024-09-15T20:32:48.737 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_service 2024-09-15T20:32:48.737 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28609 2024-09-15T20:32:48.737 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_service on pid 28609' 2024-09-15T20:32:48.737 INFO:tasks.workunit.client.0.smithi146.stdout:test api_service on pid 28609 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28609 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2>&1 | tee ceph_test_rados_api_service.log | sed "s/^/ api_service: /"' 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_service_pp 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_service_pp' 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_service_pp 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.739 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_service --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service.xml 2024-09-15T20:32:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_service.log 2024-09-15T20:32:48.740 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_service: /' 2024-09-15T20:32:48.741 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_service_pp 2024-09-15T20:32:48.741 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28626 2024-09-15T20:32:48.741 INFO:tasks.workunit.client.0.smithi146.stdout:test api_service_pp on pid 28626 2024-09-15T20:32:48.742 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_service_pp on pid 28626' 2024-09-15T20:32:48.742 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28626 2024-09-15T20:32:48.742 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.742 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.742 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2>&1 | tee ceph_test_rados_api_service_pp.log | sed "s/^/ api_service_pp: /"' 2024-09-15T20:32:48.742 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_c_write_operations 2024-09-15T20:32:48.742 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_c_write_operations' 2024-09-15T20:32:48.743 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.743 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.743 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.743 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.743 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_service_pp --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_service_pp.xml 2024-09-15T20:32:48.743 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_service_pp.log 2024-09-15T20:32:48.754 INFO:tasks.workunit.client.0.smithi146.stdout:89/0 8 ==== osd_map(20..20 src has 1..20) ==== 3534+0+0 (unknown 373424332 0 0) 0x7faec8049740 con 0x561c1df24c00 2024-09-15T20:32:48.754 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec27fc640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_map(20..20 src has 1..20) 2024-09-15T20:32:48.754 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec27fc640 3 client.4504.objecter handle_osd_map got epochs [20,20] > 0 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec27fc640 3 client.4504.objecter handle_osd_map decoding full epoch 20 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec27fc640 20 client.4504.objecter dump_active .. 0 homeless 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faed3a46880 10 librados: wait_for_osdmap done waiting 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faed3a46880 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.146:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7faec4001830 con 0x561c1df24c00 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_get_version_reply(handle=1 version=20) ==== 24+0+0 (unknown 592526333 0 0) 0x7faec8014830 con 0x561c1df24c00 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec27fc640 10 monclient: handle_get_version_reply finishing 1 version 20 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec37fe640 4 set_mon_vals no callback set 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faec37fe640 10 client.4504.objecter _get_latest_version latest 20, have it 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faed3a46880 10 client.4504.objecter create_pool name=ceph_test_rados_api_asio 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faed3a46880 10 client.4504.objecter pool_op_submit 1 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faed3a46880 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:32:48.755 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.657+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.146:6789/0 -- pool_op(create pool 0 tid 1 name ceph_test_rados_api_asio v0) -- 0x561c1ded9880 con 0x561c1df24c00 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 10 ==== pool_op_reply(tid 1 (0) Success v21) ==== 43+0+0 (unknown 3999135213 0 0) 0x7faec8015160 con 0x561c1df24c00 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 pool_op_reply(tid 1 (0) Success v21) 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 10 client.4504.objecter handle_pool_op_reply pool_op_reply(tid 1 (0) Success v21) 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 10 client.4504.objecter have request 1 at 0x561c1e076c20 Op: create 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 20 client.4504.objecter waiting for client to reach epoch 21 before calling back 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 10 client.4504.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 10 monclient: _renew_subs 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=21}) -- 0x7faeb4001d20 con 0x561c1df24c00 2024-09-15T20:32:48.756 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.745+0000 7faec27fc640 10 client.4504.objecter done 2024-09-15T20:32:48.758 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_c_write_operations 2024-09-15T20:32:48.758 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.760 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_service_pp: /' 2024-09-15T20:32:48.761 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== mon.1 v1:172.21.15.146:6789/0 11 ==== osd_map(21..21 src has 1..21) ==== 5087+0+0 (unknown 3749811291 0 0) 0x7faec80485a0 con 0x561test api_c_write_operations on pid 28639 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_c_write_operations 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28639 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_c_write_operations on pid 28639' 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28639 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2>&1 | tee ceph_test_rados_api_c_write_operations.log | sed "s/^/ api_c_write_operations: /"' 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s api_c_read_operations 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' api_c_read_operations' 2024-09-15T20:32:48.762 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo api_c_read_operations 2024-09-15T20:32:48.763 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.763 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_c_write_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_write_operations.xml 2024-09-15T20:32:48.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_c_write_operations.log 2024-09-15T20:32:48.764 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=api_c_read_operations 2024-09-15T20:32:48.765 INFO:tasks.workunit.client.0.smithi146.stdout:test api_c_read_operations on pid 28647 2024-09-15T20:32:48.765 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28647 2024-09-15T20:32:48.765 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test api_c_read_operations on pid 28647' 2024-09-15T20:32:48.765 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28647 2024-09-15T20:32:48.765 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.765 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.765 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2>&1 | tee ceph_test_rados_api_c_read_operations.log | sed "s/^/ api_c_read_operations: /"' 2024-09-15T20:32:48.765 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s list_parallel 2024-09-15T20:32:48.766 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' list_parallel' 2024-09-15T20:32:48.766 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.767 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.767 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo list_parallel 2024-09-15T20:32:48.767 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.767 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.767 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_api_c_read_operations --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/api_c_read_operations.xml 2024-09-15T20:32:48.767 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_c_read_operations: /' 2024-09-15T20:32:48.767 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_api_c_read_operations.log 2024-09-15T20:32:48.768 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ api_c_write_operations: /' 2024-09-15T20:32:48.787 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.789 INFO:tasks.workunit.client.0.smithi146.stdout:test list_parallel on pid 28674 2024-09-15T20:32:48.790 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=list_parallel 2024-09-15T20:32:48.790 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28674 2024-09-15T20:32:48.790 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test list_parallel on pid 28674' 2024-09-15T20:32:48.790 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28674 2024-09-15T20:32:48.790 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.790 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.790 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s open_pools_parallel 2024-09-15T20:32:48.790 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' open_pools_parallel' 2024-09-15T20:32:48.791 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo open_pools_parallel 2024-09-15T20:32:48.793 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.817 INFO:tasks.workunit.client.0.smithi146.stdout:test open_pools_parallel on pid 28721 2024-09-15T20:32:48.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=open_pools_parallel 2024-09-15T20:32:48.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28721 2024-09-15T20:32:48.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test open_pools_parallel on pid 28721' 2024-09-15T20:32:48.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28721 2024-09-15T20:32:48.817 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in api_aio api_aio_pp api_io api_io_pp api_asio api_list api_lock api_lock_pp api_misc api_misc_pp api_tier_pp api_pool api_snapshots api_snapshots_pp api_stat api_stat_pp api_watch_notify api_watch_notify_pp api_cmd api_cmd_pp api_service api_service_pp api_c_write_operations api_c_read_operations list_parallel open_pools_parallel delete_pools_parallel 2024-09-15T20:32:48.818 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.818 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2>&1 | tee ceph_test_rados_open_pools_parallel.log | sed "s/^/ open_pools_parallel: /"' 2024-09-15T20:32:48.818 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2>&1 | tee ceph_test_rados_list_parallel.log | sed "s/^/ list_parallel: /"' 2024-09-15T20:32:48.818 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s delete_pools_parallel 2024-09-15T20:32:48.818 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' delete_pools_parallel' 2024-09-15T20:32:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo delete_pools_parallel 2024-09-15T20:32:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ open_pools_parallel: /' 2024-09-15T20:32:48.819 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_open_pools_parallel.log 2024-09-15T20:32:48.820 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.820 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.820 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.820 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.820 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_list_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/list_parallel.xml 2024-09-15T20:32:48.821 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_list_parallel.log 2024-09-15T20:32:48.822 INFO:tasks.workunit.client.0.smithi146.stdout:test delete_pools_parallel on pid 28753 2024-09-15T20:32:48.822 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=delete_pools_parallel 2024-09-15T20:32:48.822 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28753 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test delete_pools_parallel on pid 28753' 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28753 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2>&1 | tee ceph_test_rados_delete_pools_parallel.log | sed "s/^/ delete_pools_parallel: /"' 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s cls 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ list_parallel: /' 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' cls' 2024-09-15T20:32:48.823 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_open_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/open_pools_parallel.xml 2024-09-15T20:32:48.824 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.824 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.824 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.824 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.824 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.826 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo cls 2024-09-15T20:32:48.827 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=cls 2024-09-15T20:32:48.827 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28768 2024-09-15T20:32:48.827 INFO:tasks.workunit.client.0.smithi146.stdout:test cls on pid 28768 2024-09-15T20:32:48.828 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test cls on pid 28768' 2024-09-15T20:32:48.828 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28768 2024-09-15T20:32:48.828 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.828 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.828 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s cmd 2024-09-15T20:32:48.829 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' cmd' 2024-09-15T20:32:48.830 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cls 2>&1 | tee ceph_test_neorados_cls.log | sed "s/^/ cls: /"' 2024-09-15T20:32:48.830 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.830 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo cmd 2024-09-15T20:32:48.832 INFO:tasks.workunit.client.0.smithi146.stdout:test cmd on pid 28780 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=cmd 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28780 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test cmd on pid 28780' 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28780 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_cmd 2>&1 | tee ceph_test_neorados_cmd.log | sed "s/^/ cmd: /"' 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.833 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.834 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.834 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.834 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.834 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.834 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ cls: /' 2024-09-15T20:32:48.834 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s handler_error 2024-09-15T20:32:48.836 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_cls.log 2024-09-15T20:32:48.837 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_cls 2024-09-15T20:32:48.837 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' handler_error' 2024-09-15T20:32:48.838 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo handler_error 2024-09-15T20:32:48.838 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ cmd: /' 2024-09-15T20:32:48.840 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.840 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_cmd.log 2024-09-15T20:32:48.841 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_cmd 2024-09-15T20:32:48.843 INFO:tasks.workunit.client.0.smithi146.stdout:test handler_error on pid 28799 2024-09-15T20:32:48.843 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=handler_error 2024-09-15T20:32:48.843 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28799 2024-09-15T20:32:48.843 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test handler_error on pid 28799' 2024-09-15T20:32:48.843 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28799 2024-09-15T20:32:48.843 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.843 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.844 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_handler_error 2>&1 | tee ceph_test_neorados_handler_error.log | sed "s/^/ handler_error: /"' 2024-09-15T20:32:48.844 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.845 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.845 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.845 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.861 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s io 2024-09-15T20:32:48.861 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' io' 2024-09-15T20:32:48.862 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_rados_delete_pools_parallel.log 2024-09-15T20:32:48.862 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_rados_delete_pools_parallel --gtest_output=xml:/home/ubuntu/cephtest/archive/unit_test_xml_report/delete_pools_parallel.xml 2024-09-15T20:32:48.868 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo io 2024-09-15T20:32:48.871 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_handler_error.log 2024-09-15T20:32:48.871 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ handler_error: /' 2024-09-15T20:32:48.873 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_handler_error 2024-09-15T20:32:48.873 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ delete_pools_parallel: /' 2024-09-15T20:32:48.878 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.881 INFO:tasks.workunit.client.0.smithi146.stdout:test io on pid 28865 2024-09-15T20:32:48.881 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=io 2024-09-15T20:32:48.881 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28865 2024-09-15T20:32:48.881 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test io on pid 28865' 2024-09-15T20:32:48.882 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28865 2024-09-15T20:32:48.882 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.882 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.882 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_io 2>&1 | tee ceph_test_neorados_io.log | sed "s/^/ io: /"' 2024-09-15T20:32:48.883 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.883 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.883 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.883 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.884 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_io.log 2024-09-15T20:32:48.889 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s ec_io 2024-09-15T20:32:48.889 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' ec_io' 2024-09-15T20:32:48.889 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ io: /' 2024-09-15T20:32:48.896 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo ec_io 2024-09-15T20:32:48.896 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.901 INFO:tasks.workunit.client.0.smithi146.stdout:test ec_io on pid 28928 2024-09-15T20:32:48.902 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=ec_io 2024-09-15T20:32:48.902 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28928 2024-09-15T20:32:48.902 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test ec_io on pid 28928' 2024-09-15T20:32:48.902 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28928 2024-09-15T20:32:48.902 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.902 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.902 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_io 2>&1 | tee ceph_test_neorados_ec_io.log | sed "s/^/ ec_io: /"' 2024-09-15T20:32:48.902 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s list 2024-09-15T20:32:48.903 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.903 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.903 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.903 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.903 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_io 2024-09-15T20:32:48.905 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' list' 2024-09-15T20:32:48.905 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo list 2024-09-15T20:32:48.907 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_ec_io 2024-09-15T20:32:48.907 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_ec_io.log 2024-09-15T20:32:48.907 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ ec_io: /' 2024-09-15T20:32:48.910 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.913 INFO:tasks.workunit.client.0.smithi146.stdout:test list on pid 28964 2024-09-15T20:32:48.913 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=list 2024-09-15T20:32:48.914 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28964 2024-09-15T20:32:48.914 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test list on pid 28964' 2024-09-15T20:32:48.914 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28964 2024-09-15T20:32:48.914 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.914 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.914 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_list 2>&1 | tee ceph_test_neorados_list.log | sed "s/^/ list: /"' 2024-09-15T20:32:48.915 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.915 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.915 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.915 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.915 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s ec_list 2024-09-15T20:32:48.916 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_list.log 2024-09-15T20:32:48.916 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' ec_list' 2024-09-15T20:32:48.916 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_list 2024-09-15T20:32:48.916 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo ec_list 2024-09-15T20:32:48.916 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.918 INFO:tasks.workunit.client.0.smithi146.stdout:test ec_list on pid 28995 2024-09-15T20:32:48.919 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=ec_list 2024-09-15T20:32:48.919 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ list: /' 2024-09-15T20:32:48.919 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28995 2024-09-15T20:32:48.919 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test ec_list on pid 28995' 2024-09-15T20:32:48.919 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=28995 2024-09-15T20:32:48.919 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.919 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.920 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_ec_list 2>&1 | tee ceph_test_neorados_ec_list.log | sed "s/^/ ec_list: /"' 2024-09-15T20:32:48.922 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.923 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.923 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.923 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.924 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_ec_list 2024-09-15T20:32:48.924 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s misc 2024-09-15T20:32:48.924 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' misc' 2024-09-15T20:32:48.925 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_ec_list.log 2024-09-15T20:32:48.926 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ ec_list: /' 2024-09-15T20:32:48.933 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo misc 2024-09-15T20:32:48.933 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.936 INFO:tasks.workunit.client.0.smithi146.stdout:test misc on pid 29039 2024-09-15T20:32:48.936 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=misc 2024-09-15T20:32:48.936 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29039 2024-09-15T20:32:48.936 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test misc on pid 29039' 2024-09-15T20:32:48.936 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=29039 2024-09-15T20:32:48.937 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.937 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.937 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_misc 2>&1 | tee ceph_test_neorados_misc.log | sed "s/^/ misc: /"' 2024-09-15T20:32:48.937 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s pool 2024-09-15T20:32:48.937 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' pool' 2024-09-15T20:32:48.938 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo pool 2024-09-15T20:32:48.938 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.938 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.938 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.938 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.938 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.938 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_misc 2024-09-15T20:32:48.938 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ misc: /' 2024-09-15T20:32:48.940 INFO:tasks.workunit.client.0.smithi146.stdout:test pool on pid 29059 2024-09-15T20:32:48.940 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=pool 2024-09-15T20:32:48.940 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29059 2024-09-15T20:32:48.940 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test pool on pid 29059' 2024-09-15T20:32:48.940 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=29059 2024-09-15T20:32:48.941 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.941 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.941 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_pool 2>&1 | tee ceph_test_neorados_pool.log | sed "s/^/ pool: /"' 2024-09-15T20:32:48.941 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s read_operations 2024-09-15T20:32:48.942 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' read_operations' 2024-09-15T20:32:48.942 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.942 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.942 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.942 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.942 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_pool 2024-09-15T20:32:48.942 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_pool.log 2024-09-15T20:32:48.943 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ pool: /' 2024-09-15T20:32:48.943 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.944 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_misc.log 2024-09-15T20:32:48.947 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo read_operations 2024-09-15T20:32:48.948 INFO:tasks.workunit.client.0.smithi146.stdout:test read_operations on pid 29093 2024-09-15T20:32:48.948 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=read_operations 2024-09-15T20:32:48.948 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29093 2024-09-15T20:32:48.948 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test read_operations on pid 29093' 2024-09-15T20:32:48.948 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=29093 2024-09-15T20:32:48.948 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.949 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.951 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_read_operations 2>&1 | tee ceph_test_neorados_read_operations.log | sed "s/^/ read_operations: /"' 2024-09-15T20:32:48.951 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s snapshots 2024-09-15T20:32:48.952 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' snapshots' 2024-09-15T20:32:48.953 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.954 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.954 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.954 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.954 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo snapshots 2024-09-15T20:32:48.954 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_read_operations 2024-09-15T20:32:48.954 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ read_operations: /' 2024-09-15T20:32:48.954 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.955 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_read_operations.log 2024-09-15T20:32:48.957 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=snapshots 2024-09-15T20:32:48.957 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29124 2024-09-15T20:32:48.957 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test snapshots on pid 29124' 2024-09-15T20:32:48.958 INFO:tasks.workunit.client.0.smithi146.stdout:test snapshots on pid 29124 2024-09-15T20:32:48.958 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=29124 2024-09-15T20:32:48.958 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.959 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.959 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_snapshots 2>&1 | tee ceph_test_neorados_snapshots.log | sed "s/^/ snapshots: /"' 2024-09-15T20:32:48.959 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s watch_notify 2024-09-15T20:32:48.959 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' watch_notify' 2024-09-15T20:32:48.960 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.960 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.960 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.960 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.960 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_snapshots 2024-09-15T20:32:48.960 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_snapshots.log 2024-09-15T20:32:48.960 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ snapshots: /' 2024-09-15T20:32:48.965 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo watch_notify 2024-09-15T20:32:48.965 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.968 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=watch_notify 2024-09-15T20:32:48.968 INFO:tasks.workunit.client.0.smithi146.stdout:test watch_notify on pid 29152 2024-09-15T20:32:48.969 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29152 2024-09-15T20:32:48.969 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test watch_notify on pid 29152' 2024-09-15T20:32:48.969 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=29152 2024-09-15T20:32:48.969 INFO:tasks.workunit.client.0.smithi146.stderr:+ for f in cls cmd handler_error io ec_io list ec_list misc pool read_operations snapshots watch_notify write_operations 2024-09-15T20:32:48.969 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.970 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_watch_notify 2>&1 | tee ceph_test_neorados_watch_notify.log | sed "s/^/ watch_notify: /"' 2024-09-15T20:32:48.970 INFO:tasks.workunit.client.0.smithi146.stderr:++ printf %25s write_operations 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:+ r=' write_operations' 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:++ echo write_operations 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:++ awk '{print $1}' 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_watch_notify 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ watch_notify: /' 2024-09-15T20:32:48.971 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_watch_notify.log 2024-09-15T20:32:48.973 INFO:tasks.workunit.client.0.smithi146.stdout:test write_operations on pid 29169 2024-09-15T20:32:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:+ ff=write_operations 2024-09-15T20:32:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29169 2024-09-15T20:32:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:+ echo 'test write_operations on pid 29169' 2024-09-15T20:32:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:+ pids[$f]=29169 2024-09-15T20:32:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:+ bash -o pipefail -exc 'ceph_test_neorados_write_operations 2>&1 | tee ceph_test_neorados_write_operations.log | sed "s/^/ write_operations: /"' 2024-09-15T20:32:48.974 INFO:tasks.workunit.client.0.smithi146.stderr:+ ret=0 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' 1 -eq 1 ']' 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28316 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28316 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ '[' -z '' ']' 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ case $- in 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ return 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ ceph_test_neorados_write_operations 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ sed 's/^/ write_operations: /' 2024-09-15T20:32:48.975 INFO:tasks.workunit.client.0.smithi146.stderr:+ tee ceph_test_neorados_write_operations.log 2024-09-15T20:32:49.915 INFO:tasks.workunit.client.0.smithi146.stdout:c1df24c00 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faec27fc640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_map(21..21 src has 1..21) 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faec27fc640 3 client.4504.objecter handle_osd_map got epochs [21,21] > 20 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faec27fc640 3 client.4504.objecter handle_osd_map decoding incremental epoch 21 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faec27fc640 20 client.4504.objecter dump_active .. 0 homeless 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 10 librados: set snap read head -> 1 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 10 librados: writefull oid=exist nspace= 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0888b0 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 21 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:49.916 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 2 osd.0 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 2 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 15 client.4504.objecter _send_op 2 to 9.1 on osd.0 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:2 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e21) -- 0x561c1e08cd80 con 0x561c1e089270 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faed1d51640 10 monclient: get_auth_request con 0x561c1e089270 auth_method 0 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:48.749+0000 7faec27fc640 10 client.4504.objecter ms_handle_connect 0x561c1e089270 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 1 ==== osd_map(22..22 src has 1..22) ==== 13178+0+0 (unknown 3518016485 0 0) 0x7faec40040f0 con 0x561c1e089270 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_map(22..22 src has 1..22) 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 3 client.4504.objecter handle_osd_map got epochs [22,22] > 21 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 3 client.4504.objecter handle_osd_map decoding incremental epoch 22 2024-09-15T20:32:49.917 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 10 client.4504.objecter checking op 2 2024-09-15T20:32:49.918 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:32:49.918 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:49.918 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 20 client.4504.objecter dump_active .. 0 homeless 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.909+0000 7faec27fc640 20 client.4504.objecter 2 9.d3a59201 osd.0 exist [writefull 0~5 in=5b] 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 2 ==== osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) ==== 149+0+0 (unknown 197017736 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(2 exist [writefull 0~5] v22'1 uv1 ondisk = 0) 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 2 ondisk uv 1 in 9.1 attempt 0 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 2 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 15 client.4504.objecter _finish_op 2 2024-09-15T20:32:49.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 2 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncReadCallback 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 10 librados: Objecter returned from writefull r=0 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0878a0 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 3 osd.0 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 3 2024-09-15T20:32:49.987 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 15 client.4504.objecter _send_op 3 to 9.1 on osd.0 2024-09-15T20:32:49.988 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:3 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e087d50 con 0x561c1e089270 2024-09-15T20:32:49.988 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:49.988 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:49.988 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:49.988 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:49.988 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-09-15T20:32:49.988 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.002 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.002 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 10 client.4504.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 4 osd.0 2024-09-15T20:32:50.002 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 4 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 15 client.4504.objecter _send_op 4 to 9.1 on osd.0 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:4 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e0885a0 con 0x561c1e089270 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.981+0000 7faed3a46880 5 client.4504.objecter 2 in flight 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 3 ==== osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 1950226522 0 3741568377) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(3 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 3 ondisk uv 1 in 9.1 attempt 0 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 5 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 3 2024-09-15T20:32:50.003 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter _finish_op 3 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 3 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 4 ==== osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3823496323 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(4 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 4 ondisk uv 0 in 9.1 attempt 0 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 4 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter _finish_op 4 2024-09-15T20:32:50.004 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.005 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 4 2024-09-15T20:32:50.005 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.005 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncReadCallback (17 ms) 2024-09-15T20:32:50.005 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncReadFuture 2024-09-15T20:32:50.005 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:50.005 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 5 osd.0 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 5 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 15 client.4504.objecter _send_op 5 to 9.1 on osd.0 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:5 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e088940 con 0x561c1e089270 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0878a0 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-09-15T20:32:50.006 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 6 osd.0 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 6 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 15 client.4504.objecter _send_op 6 to 9.1 on osd.0 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:6 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08d120 con 0x561c1e089270 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 5 client.4504.objecter 2 in flight 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 5 ==== osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 1950226522 0 3741568377) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(5 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 5 ondisk uv 1 in 9.1 attempt 0 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 5 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 5 2024-09-15T20:32:50.007 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter _finish_op 5 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 5 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 6 ==== osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3823496323 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(6 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 6 ondisk uv 0 in 9.1 attempt 0 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 6 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter _finish_op 6 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.008 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 6 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncReadFuture (2 ms) 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncReadYield 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [read 0~256] tid 7 osd.0 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 7 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 15 client.4504.objecter _send_op 7 to 9.1 on osd.0 2024-09-15T20:32:50.009 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:7 9.1 9:8049a5cb:::exist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e0879b0 con 0x561c1e089270 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0cd640 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 10 client.4504.objecter _op_submit oid noexist '@9' '@9' [read 0~256] tid 8 osd.0 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 8 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 15 client.4504.objecter _send_op 8 to 9.1 on osd.0 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:8 9.1 9:855dde50:::noexist:head [read 0~256] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e0cdaf0 con 0x561c1e089270 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.010 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:49.997+0000 7faed3a46880 5 client.4504.objecter 2 in flight 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 7 ==== osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) ==== 149+0+5 (unknown 1950226522 0 3741568377) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(7 exist [read 0~5 out=5b] v0'0 uv1 ondisk = 0) 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 7 ondisk uv 1 in 9.1 attempt 0 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 5 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 7 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 15 client.4504.objecter _finish_op 7 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 7 2024-09-15T20:32:50.011 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 8 ==== osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 3823496323 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(8 noexist [read 0~256] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 8 ondisk uv 0 in 9.1 attempt 0 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 10 client.4504.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 8 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 15 client.4504.objecter _finish_op 8 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 8 2024-09-15T20:32:50.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncReadYield (0 ms) 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteCallback 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 20 librados: queue_aio_write 0x561c1de9e530 completion 0x561c1e0cd640 write_seq 1 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 9 osd.0 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 9 2024-09-15T20:32:50.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 15 client.4504.objecter _send_op 9 to 9.1 on osd.0 2024-09-15T20:32:50.028 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:9 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x561c1e0cd750 con 0x561c1e089270 2024-09-15T20:32:50.028 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.028 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.028 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.001+0000 7faed3a46880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T20:32:50.028 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 9 ==== osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) ==== 149+0+0 (unknown 185088608 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.032 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(9 exist [write 0~5] v22'2 uv2 ondisk = 0) 2024-09-15T20:32:50.032 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.032 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 9 ondisk uv 2 in 9.1 attempt 0 2024-09-15T20:32:50.032 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.032 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 9 2024-09-15T20:32:50.032 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 15 client.4504.objecter _finish_op 9 2024-09-15T20:32:50.032 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 9 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed1d51640 20 librados: complete_aio_write 0x561c1e0cd640 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncWriteCallback (21 ms) 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteFuture 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 20 librados: queue_aio_write 0x561c1de9e530 completion 0x561c1e08e5e0 write_seq 2 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.033 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 10 osd.0 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 10 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 15 client.4504.objecter _send_op 10 to 9.1 on osd.0 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:10 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08e710 con 0x561c1e089270 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.021+0000 7faed3a46880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 10 ==== osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) ==== 149+0+0 (unknown 2139652532 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(10 exist [write 0~5] v22'3 uv3 ondisk = 0) 2024-09-15T20:32:50.034 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.035 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 10 ondisk uv 3 in 9.1 attempt 0 2024-09-15T20:32:50.035 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 10 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 15 client.4504.objecter _finish_op 10 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 10 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed1d51640 20 librados: complete_aio_write 0x561c1e08e5e0 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncWriteFuture (7 ms) 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteYield 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=head 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 20 librados: queue_aio_write 0x561c1de9e530 completion 0x561c1e08e710 write_seq 3 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:32:50.036 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [write 0~5 in=5b] tid 11 osd.0 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 11 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 15 client.4504.objecter _send_op 11 to 9.1 on osd.0 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:11 9.1 9:8049a5cb:::exist:head [write 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x561c1e0b0c20 con 0x561c1e089270 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.025+0000 7faed3a46880 20 librados: aio_write exist 0~5 snapc=0=[] snap_seq=1 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 11 ==== osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) ==== 149+0+0 (unknown 904385641 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.037 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(11 exist [write 0~5] v22'4 uv4 ondisk = 0) 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 11 ondisk uv 4 in 9.1 attempt 0 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 11 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 15 client.4504.objecter _finish_op 11 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 11 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed1d51640 20 librados: complete_aio_write 0x561c1e08e710 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncWriteYield (3 ms) 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationCallback 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.038 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 12 osd.0 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 12 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 15 client.4504.objecter _send_op 12 to 9.1 on osd.0 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:12 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08d4c0 con 0x561c1e089270 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e08d9e0 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-09-15T20:32:50.039 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.040 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.040 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 10 client.4504.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 13 osd.0 2024-09-15T20:32:50.040 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.040 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 13 2024-09-15T20:32:50.040 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 15 client.4504.objecter _send_op 13 to 9.1 on osd.0 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:13 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08de90 con 0x561c1e089270 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.029+0000 7faed3a46880 5 client.4504.objecter 2 in flight 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 12 ==== osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 107805355 0 3741568377) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(12 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 12 ondisk uv 4 in 9.1 attempt 0 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 5 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 12 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _finish_op 12 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 12 2024-09-15T20:32:50.042 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 13 ==== osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 2389562043 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(13 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationCallback (6 ms) 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationFuture 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 13 ondisk uv 0 in 9.1 attempt 0 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 13 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _finish_op 13 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 13 2024-09-15T20:32:50.043 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.044 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:50.044 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.044 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.044 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.044 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.044 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 14 osd.0 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 14 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _send_op 14 to 9.1 on osd.0 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:14 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08d4c0 con 0x561c1e089270 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e08d9c0 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 15 osd.0 2024-09-15T20:32:50.045 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 15 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _send_op 15 to 9.1 on osd.0 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:15 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08de70 con 0x561c1e089270 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 5 client.4504.objecter 2 in flight 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 14 ==== osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 107805355 0 3741568377) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(14 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 14 ondisk uv 4 in 9.1 attempt 0 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 5 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 14 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _finish_op 14 2024-09-15T20:32:50.046 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 14 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 15 ==== osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 2389562043 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(15 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 15 ondisk uv 0 in 9.1 attempt 0 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 15 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _finish_op 15 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 15 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.047 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationFuture (1 ms) 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncReadOperationYield 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e0880f0 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [read 0~0] tid 16 osd.0 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 16 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _send_op 16 to 9.1 on osd.0 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:16 9.1 9:8049a5cb:::exist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08d4c0 con 0x561c1e089270 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.048 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e08d9c0 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base noexist @9 precalc_pgid 0 pgid 0.0 is_read 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target target noexist @9 -> pgid 9.a7bbaa1 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.a7bbaa1 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit oid noexist '@9' '@9' [read 0~0] tid 17 osd.0 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 17 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _send_op 17 to 9.1 on osd.0 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:17 9.1 9:855dde50:::noexist:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e22) -- 0x561c1e0b0c20 con 0x561c1e089270 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 5 client.4504.objecter 2 in flight 2024-09-15T20:32:50.049 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 16 ==== osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) ==== 149+0+5 (unknown 107805355 0 3741568377) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(16 exist [read 0~5 out=5b] v0'0 uv4 ondisk = 0) 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 16 ondisk uv 4 in 9.1 attempt 0 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 5 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 16 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _finish_op 16 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 5 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 16 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 17 ==== osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) ==== 151+0+0 (unknown 2389562043 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(17 noexist [read 0~0] v0'0 uv0 ondisk = -2 ((2) No such file or directory)) 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncReadOperationYield (1 ms) 2024-09-15T20:32:50.050 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 17 ondisk uv 0 in 9.1 attempt 0 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationCallback 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter handle_osd_op_reply copying resulting 0 into existing ceph::buffer of length 0 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 17 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _finish_op 17 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 17 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.051 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 librados: queue_aio_write 0x561c1de9e530 completion 0x561c1e08e400 write_seq 4 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e08d9c0 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 18 osd.0 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 18 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 15 client.4504.objecter _send_op 18 to 9.1 on osd.0 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:18 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08d4c0 con 0x561c1e089270 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.037+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.052 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 18 ==== osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) ==== 149+0+0 (unknown 3696285993 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(18 exist [writefull 0~5] v22'5 uv5 ondisk = 0) 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 18 ondisk uv 5 in 9.1 attempt 0 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 18 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 15 client.4504.objecter _finish_op 18 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 18 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.053 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 20 librados: complete_aio_write 0x561c1e08e400 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationCallback (5 ms) 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationFuture 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 librados: queue_aio_write 0x561c1de9e530 completion 0x561c1e08df10 write_seq 5 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e08d9c0 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 19 osd.0 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 19 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 15 client.4504.objecter _send_op 19 to 9.1 on osd.0 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:19 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08d4c0 con 0x561c1e089270 2024-09-15T20:32:50.054 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 19 ==== osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) ==== 149+0+0 (unknown 1104221525 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(19 exist [writefull 0~5] v22'6 uv6 ondisk = 0) 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 19 ondisk uv 6 in 9.1 attempt 0 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 19 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 15 client.4504.objecter _finish_op 19 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 19 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed1d51640 20 librados: complete_aio_write 0x561c1e08df10 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationFuture (4 ms) 2024-09-15T20:32:50.055 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ RUN ] AsioRados.AsyncWriteOperationYield 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 librados: queue_aio_write 0x561c1de9e530 completion 0x561c1e08e0d0 write_seq 6 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 10 client.4504.objecter _op_submit op 0x561c1e08d9c0 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter _calc_target epoch 22 base exist @9 precalc_pgid 0 pgid 0.0 is_write 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter _calc_target target exist @9 -> pgid 9.d3a59201 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 10 client.4504.objecter _calc_target raw pgid 9.d3a59201 -> actual 9.1 acting [0,2] primary 0 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter _get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 10 client.4504.objecter _op_submit oid exist '@9' '@9' [writefull 0~5 in=5b] tid 20 osd.0 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter get_session s=0x561c1e057c30 osd=0 3 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 15 client.4504.objecter _session_op_assign 0 20 2024-09-15T20:32:50.058 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 15 client.4504.objecter _send_op 20 to 9.1 on osd.0 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 --> v1:172.21.15.78:6804/3648339548 -- osd_op(unknown.0.0:20 9.1 9:8049a5cb:::exist:head [writefull 0~5 in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22) -- 0x561c1e08d4c0 con 0x561c1e089270 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.045+0000 7faed3a46880 5 client.4504.objecter 1 in flight 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 <== osd.0 v1:172.21.15.78:6804/3648339548 20 ==== osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) ==== 149+0+0 (unknown 894545537 0 0) 0x7faec40043f0 con 0x561c1e089270 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 10 client.4504.objecter ms_dispatch 0x561c1dea8f00 osd_op_reply(20 exist [writefull 0~5] v22'7 uv7 ondisk = 0) 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 10 client.4504.objecter in handle_osd_op_reply 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 7 client.4504.objecter handle_osd_op_reply 20 ondisk uv 7 in 9.1 attempt 0 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 10 client.4504.objecter op 0 rval 0 len 0 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 15 client.4504.objecter handle_osd_op_reply completed tid 20 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 15 client.4504.objecter _finish_op 20 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 4 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 15 client.4504.objecter _session_op_remove 0 20 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 5 client.4504.objecter 0 in flight 2024-09-15T20:32:50.059 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 20 librados: complete_aio_write 0x561c1e08e0d0 2024-09-15T20:32:50.060 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ OK ] AsioRados.AsyncWriteOperationYield (5 ms) 2024-09-15T20:32:50.060 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 10 librados: watch_flush enter 2024-09-15T20:32:50.060 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 10 librados: watch_flush exit 2024-09-15T20:32:50.060 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 10 client.4504.objecter close_session for osd.0 2024-09-15T20:32:50.060 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 >> v1:172.21.15.78:6804/3648339548 conn(0x561c1e089270 legacy=0x561c1e08b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 20 client.4504.objecter put_session s=0x561c1e057c30 osd=0 1 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 20 client.4504.objecter shutdown clearing up homeless session... 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 10 client.4504.objecter successfully canceled tick 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 >> v1:172.21.15.146:6812/435598642 conn(0x7faeb403c430 legacy=0x7faeb403e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 10 monclient: shutdown 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 >> v1:172.21.15.146:6789/0 conn(0x561c1df24c00 legacy=0x561c1e07df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed1d51640 1 -- 172.21.15.146:0/1037512150 reap_dead start 2024-09-15T20:32:50.062 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 shutdown_connections 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 >> 172.21.15.146:0/1037512150 conn(0x561c1df1ce60 msgr2=0x561c1df1d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 shutdown_connections 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 1 -- 172.21.15.146:0/1037512150 wait complete. 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.053+0000 7faed3a46880 1 librados: shutdown 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [----------] 12 tests from AsioRados (72 ms total) 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [----------] Global test environment tear-down 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [==========] 12 tests from 1 test suite ran. (1412 ms total) 2024-09-15T20:32:50.063 INFO:tasks.workunit.client.0.smithi146.stdout: api_asio: [ PASSED ] 12 tests. 2024-09-15T20:32:50.453 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [==========] Running 11 tests from 3 test suites. 2024-09-15T20:32:50.453 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [----------] Global test environment set-up. 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [----------] 7 tests from LibRadosList 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosList.ListObjects 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosList.ListObjects (251 ms) 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosList.ListObjectsZeroInName 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosList.ListObjectsZeroInName (32 ms) 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosList.ListObjectsNS 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: myset foo1,foo2,foo3 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo1 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo2 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo3 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: myset foo1,foo4,foo5 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo4 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo5 2024-09-15T20:32:50.454 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo1 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: myset foo6,foo7 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo7 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo6 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns1:foo4 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns1:foo5 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns2:foo7 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns2:foo6 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns1:foo1 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: :foo1 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: :foo2 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: :foo3 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosList.ListObjectsNS (91 ms) 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosList.ListObjectsStart 2024-09-15T20:32:50.455 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 1 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 10 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 13 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 7 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 14 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 0 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 15 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 11 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 5 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 8 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 6 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 3 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 4 0 2024-09-15T20:32:50.456 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 12 0 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 9 0 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 2 0 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosList.ListObjectsStart (144 ms) 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosList.ListObjectsCursor 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: x cursor=MIN 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=1 cursor=10:02547ec2:::1:head 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=10 cursor=10:52ea6a34:::10:head 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=13 cursor=10:566253c9:::13:head 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=7 cursor=10:5c6b0b28:::7:head 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=14 cursor=10:62a1935d:::14:head 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=0 cursor=10:6cac518f:::0:head 2024-09-15T20:32:50.457 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=15 cursor=10:863748b0:::15:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=11 cursor=10:89d3ae78:::11:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=5 cursor=10:b29083e3:::5:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=8 cursor=10:bd63b0f1:::8:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=6 cursor=10:c4fdafeb:::6:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=3 cursor=10:cfc208b3:::3:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=4 cursor=10:d83876eb:::4:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=12 cursor=10:de5d7c5f:::12:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=9 cursor=10:e960b815:::9:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > oid=2 cursor=10:f905c69b:::2:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: FIRST> seek to MIN oid=1 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=1 cursor=10:02547ec2:::1:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-09-15T20:32:50.458 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=10 cursor=10:52ea6a34:::10:head 2024-09-15T20:32:50.459 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-09-15T20:32:50.459 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-09-15T20:32:50.459 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=13 cursor=10:566253c9:::13:head 2024-09-15T20:32:50.459 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:566253c9:::13:head 2024-09-15T20:32:50.459 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-09-15T20:32:50.459 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=7 cursor=10:5c6b0b28:::7:head 2024-09-15T20:32:50.459 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-09-15T20:32:50.459 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-09-15T20:32:50.564 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=14 cursor=10:62a1935d:::14:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=0 cursor=10:6cac518f:::0:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=15 cursor=10:863748b0:::15:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:863748b0:::15:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=11 cursor=10:89d3ae78:::11:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=5 cursor=10:b29083e3:::5:head 2024-09-15T20:32:50.565 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=8 cursor=10:bd63b0f1:::8:head 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=6 cursor=10:c4fdafeb:::6:head 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=3 cursor=10:cfc208b3:::3:head 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-09-15T20:32:50.566 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=4 cursor=10:d83876eb:::4:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=12 cursor=10:de5d7c5f:::12:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=9 cursor=10:e960b815:::9:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:e960b815:::9:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : oid=2 cursor=10:f905c69b:::2:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:d83876eb:::4:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:d83876eb:::4:head expected=10:d83876eb:::4:head 2024-09-15T20:32:50.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:d83876eb:::4:head -> 4 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=4 expected=4 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:cfc208b3:::3:head 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:cfc208b3:::3:head expected=10:cfc208b3:::3:head 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:cfc208b3:::3:head -> 3 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=3 expected=3 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:c4fdafeb:::6:head 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:c4fdafeb:::6:head expected=10:c4fdafeb:::6:head 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:c4fdafeb:::6:head -> 6 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=6 expected=6 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:bd63b0f1:::8:head 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:bd63b0f1:::8:head expected=10:bd63b0f1:::8:head 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:bd63b0f1:::8:head -> 8 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=8 expected=8 2024-09-15T20:32:50.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:b29083e3:::5:head 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:b29083e3:::5:head expected=10:b29083e3:::5:head 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:b29083e3:::5:head -> 5 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=5 expected=5 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:89d3ae78:::11:head 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:89d3ae78:::11:head expected=10:89d3ae78:::11:head 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:89d3ae78:::11:head -> 11 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=11 expected=11 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:5c6b0b28:::7:head 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:5c6b0b28:::7:head expected=10:5c6b0b28:::7:head 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:5c6b0b28:::7:head -> 7 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=7 expected=7 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:f905c69b:::2:head 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:f905c69b:::2:head expected=10:f905c69b:::2:head 2024-09-15T20:32:50.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:f905c69b:::2:head -> 2 2024-09-15T20:32:51.798 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=2 expect handler_error: Running main() from gmock_main.cc 2024-09-15T20:32:51.798 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [==========] Running 1 test from 1 test suite. 2024-09-15T20:32:51.798 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [----------] Global test environment set-up. 2024-09-15T20:32:51.798 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [----------] 1 test from neocls_handler_error 2024-09-15T20:32:51.798 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [ RUN ] neocls_handler_error.test_handler_error 2024-09-15T20:32:51.799 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [ OK ] neocls_handler_error.test_handler_error (2908 ms) 2024-09-15T20:32:51.799 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [----------] 1 test from neocls_handler_error (2908 ms total) 2024-09-15T20:32:51.799 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: 2024-09-15T20:32:51.799 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [----------] Global test environment tear-down 2024-09-15T20:32:51.799 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [==========] 1 test from 1 test suite ran. (2908 ms total) 2024-09-15T20:32:51.799 INFO:tasks.workunit.client.0.smithi146.stdout: handler_error: [ PASSED ] 1 test. 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: Running main() from gmock_main.cc 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [==========] Running 3 tests from 1 test suite. 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [----------] Global test environment set-up. 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.MonDescribePP 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [ OK ] LibRadosCmd.MonDescribePP (85 ms) 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.OSDCmdPP 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [ OK ] LibRadosCmd.OSDCmdPP (52 ms) 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [ RUN ] LibRadosCmd.PGCmdPP 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [ OK ] LibRadosCmd.PGCmdPP (2913 ms) 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [----------] 3 tests from LibRadosCmd (3050 ms total) 2024-09-15T20:32:51.803 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [----------] Global test environment tear-down 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [==========] 3 tests from 1 test suite ran. (3050 ms total) 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd_pp: [ PASSED ] 3 tests. 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: cls: Running main() from gmock_main.cc 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [==========] Running 1 test from 1 test suite. 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [----------] Global test environment set-up. 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [----------] 1 test from NeoRadosCls 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [ RUN ] NeoRadosCls.DNE 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [ OK ] NeoRadosCls.DNE (2945 ms) 2024-09-15T20:32:51.804 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [----------] 1 test from NeoRadosCls (2945 ms total) 2024-09-15T20:32:51.805 INFO:tasks.workunit.client.0.smithi146.stdout: cls: 2024-09-15T20:32:51.805 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [----------] Global test environment tear-down 2024-09-15T20:32:51.805 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [==========] 1 test from 1 test suite ran. (2945 ms total) 2024-09-15T20:32:51.805 INFO:tasks.workunit.client.0.smithi146.stdout: cls: [ PASSED ] 1 test. 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: Running main() from gmock_main.cc 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [==========] Running 17 tests from 1 test suite. 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [----------] Global test environment set-up. 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.NewDelete 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.NewDelete (0 ms) 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.SetOpFlags 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.SetOpFlags (247 ms) 2024-09-15T20:32:51.813 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertExists 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertExists (13 ms) 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.AssertVersion 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.AssertVersion (23 ms) 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpXattr 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpXattr (31 ms) 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Read 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Read (9 ms) 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Checksum 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Checksum (8 ms) 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.RWOrderedRead 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.RWOrderedRead (12 ms) 2024-09-15T20:32:51.814 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ShortRead 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ShortRead (11 ms) 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Exec 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Exec (11 ms) 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.ExecUserBuf 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.ExecUserBuf (11 ms) 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat (6 ms) 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Stat2 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Stat2 (6 ms) 2024-09-15T20:32:51.815 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.Omap 2024-09-15T20:32:51.816 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.Omap (32 ms) 2024-09-15T20:32:51.816 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.OmapNuls 2024-09-15T20:32:51.816 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.OmapNuls (24 ms) 2024-09-15T20:32:51.816 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.GetXattrs 2024-09-15T20:32:51.816 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.GetXattrs (22 ms) 2024-09-15T20:32:51.816 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ RUN ] CReadOpsTest.CmpExt 2024-09-15T20:32:51.816 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ OK ] CReadOpsTest.CmpExt (5 ms) 2024-09-15T20:32:51.817 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [----------] 17 tests from CReadOpsTest (471 ms total) 2024-09-15T20:32:51.817 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: 2024-09-15T20:32:51.817 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [----------] Global test environment tear-down 2024-09-15T20:32:51.817 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [==========] 17 tests from 1 test suite ran. (3029 ms total) 2024-09-15T20:32:51.817 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_read_operations: [ PASSED ] 17 tests. 2024-09-15T20:32:51.962 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: Running main() from gmock_main.cc 2024-09-15T20:32:51.962 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [==========] Running 4 tests from 1 test suite. 2024-09-15T20:32:51.962 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [----------] Global test environment set-up. 2024-09-15T20:32:51.962 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [----------] 4 tests from LibRadosCmd 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ RUN ] LibRadosCmd.MonDescribe 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ OK ] LibRadosCmd.MonDescribe (79 ms) 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ RUN ] LibRadosCmd.OSDCmd 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ OK ] LibRadosCmd.OSDCmd (87 ms) 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ RUN ] LibRadosCmd.PGCmd 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ OK ] LibRadosCmd.PGCmd (2898 ms) 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ RUN ] LibRadosCmd.WatchLog 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.768006+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/189437739' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28584-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.768182+0000 mon.a [INF] from='client.4555 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStat_smithi146-28423-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.768380+0000 mon.a [INF] from='client.4558 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosStatPP_smithi146-28435-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.768537+0000 mon.a [INF] from='client.4654 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28604-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.768722+0000 mon.a [INF] from='client.4567 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "LibRadosWatchNotify_smithi146-28475-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:51.963 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.768845+0000 mon.a [INF] from='client.4591 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "CReadOpsTest_smithi146-28652-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.769043+0000 mon.a [INF] from='client.4510 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi146-28209-1","var":"pg_num","val":"11"}]': finished 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.777998+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2196099040' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi146-28209-1","var":"pgp_num","val":"11"} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.800470+0000 mon.a [WRN] Health check failed: 22 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.804528+0000 mon.a [INF] from='client.4510 ' entity='client.admin' cmd={"prefix":"osd pool set","pool":"LibRadosList_smithi146-28209-1","var":"pgp_num","val":"11"} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.807553+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/1732815460' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28320-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.899300+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/609894408' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.906038+0000 mon.b [INF] from='client.4750 v1:172.21.15.146:0/609894408' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.927627+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.945284+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:50.963670+0000 mon.b [INF] from='client.4750 v1:172.21.15.146:0/609894408' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.024819+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:51.964 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.025873+0000 mon.b [INF] from='client.4750 v1:172.21.15.146:0/609894408' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi146-28185-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.037955+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi146-28185-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.774167+0000 mon.a [INF] from='client.4678 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "SimpleWritesmithi146-28936-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-SimpleWritesmithi146-28936-1"}]': finished 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.774279+0000 mon.a [INF] from='client.4684 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "ListObjectssmithi146-29018-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectssmithi146-29018-1"}]': finished 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.774347+0000 mon.a [INF] from='client.4570 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosWatchNotifyECPP_smithi146-28502-1", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosWatchNotifyECPP_smithi146-28502-1"}]': finished 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.774437+0000 mon.a [INF] from='client.4510 ' entity='client.admin' cmd='[{"prefix":"osd pool set","pool":"LibRadosList_smithi146-28209-1","var":"pgp_num","val":"11"}]': finished 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.774621+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/1732815460' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28320-4","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.774763+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoEC_smithi146-28185-16", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.816295+0000 mon.b [INF] from='client.4750 v1:172.21.15.146:0/609894408' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi146-28185-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:51.965 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.818178+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi146-28185-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:52.125 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.819299+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/4221173951' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "Simp list_parallel: process_1_[28792]: starting. 2024-09-15T20:32:52.125 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_1_[28792]: creating pool ceph_test_rados_list_parallel.smithi146-28745 2024-09-15T20:32:52.125 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_1_[28792]: created object 0... 2024-09-15T20:32:52.125 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_1_[28792]: created object 25... 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_1_[28792]: created object 49... 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_1_[28792]: finishing. 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_1_[28792]: shutting down. 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_2_[28793]: starting. 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_2_[28793]: listing objects. 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_2_[28793]: listed object 0... 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_2_[28793]: listed object 25... 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_2_[28793]: saw 50 objects 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_2_[28793]: shutting down. 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_3_[29385]: starting. 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_3_[29385]: creating pool ceph_test_rados_list_parallel.smithi146-28745 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_3_[29385]: created object 0... 2024-09-15T20:32:52.126 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_3_[29385]: created object 25... 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_3_[29385]: created object 49... 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_3_[29385]: finishing. 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_3_[29385]: shutting down. 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_4_[29386]: starting. 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_4_[29386]: listing objects. 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_4_[29386]: listed object 0... 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_4_[29386]: listed object 25... 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_4_[29386]: saw 44 objects 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_4_[29386]: shutting down. 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_5_[29387]: starting. 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_5_[29387]: removed 25 objects... 2024-09-15T20:32:52.127 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_5_[29387]: removed half of the objects 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_5_[29387]: removed 50 objects... 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_5_[29387]: removed 50 objects 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_5_[29387]: shutting down. 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_6_[29436]: starting. 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_6_[29436]: creating pool ceph_test_rados_list_parallel.smithi146-28745 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_6_[29436]: created object 0... 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_6_[29436]: created object 25... 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_6_[29436]: created object 49... 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_6_[29436]: finishing. 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_6_[29436]: shutting down. 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.128 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_7_[29437]: starting. 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_7_[29437]: listing objects. 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_7_[29437]: listed object 0... 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_7_[29437]: listed object 25... 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_7_[29437]: listed object 50... 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_7_[29437]: saw 56 objects 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_7_[29437]: shutting down. 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_8_[29438]: starting. 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_8_[29438]: added 25 objects... 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_8_[29438]: added half of the objects 2024-09-15T20:32:52.129 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_8_[29438]: added 50 objects... 2024-09-15T20:32:52.130 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_8_[29438]: added 50 objects 2024-09-15T20:32:52.130 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_8_[29438]: shutting down. 2024-09-15T20:32:52.130 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.130 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.462 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.462 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_9_[29487]: starting. 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_9_[29487]: creating pool ceph_test_rados_list_parallel.smithi146-28745 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_9_[29487]: created object 0... 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_9_[29487]: created object 25... 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_9_[29487]: created object 49... 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_9_[29487]: finishing. 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_9_[29487]: shutting down. 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_10_[29488]: starting. 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_10_[29488]: listing objects. 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_10_[29488]: listed object 0... 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_10_[29488]: listed object 25... 2024-09-15T20:32:52.463 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_10_[29488]: listed object 50... 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_10_[29488]: listed object 75... 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_10_[29488]: saw 99 objects 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_10_[29488]: shutting down. 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_13_[29491]: starting. 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_13_[29491]: removed 25 objects... 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_13_[29491]: removed half of the objects 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_13_[29491]: removed 50 objects... 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_13_[29491]: removed 50 objects 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_13_[29491]: shutting down. 2024-09-15T20:32:52.464 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_12_[29490]: starting. 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_12_[29490]: added 25 objects... 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_12_[29490]: added half of the objects 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_12_[29490]: added 50 objects... 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_12_[29490]: added 50 objects 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_12_[29490]: shutting down. 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_11_[29489]: starting. 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_11_[29489]: added 25 objects... 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_11_[29489]: added half of the objects 2024-09-15T20:32:52.465 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_11_[29489]: added 50 objects... 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_11_[29489]: added 50 objects 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_11_[29489]: shutting down. 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_14_[29802]: starting. 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_14_[29802]: creating pool ceph_test_rados_list_parallel.smithi146-28745 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_14_[29802]: created object 0... 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_14_[29802]: created object 25... 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_14_[29802]: created object 49... 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_14_[29802]: finishing. 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_14_[29802]: shutting down. 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.466 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.467 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.467 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.467 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: starting. 2024-09-15T20:32:52.467 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: listing objects. 2024-09-15T20:32:52.467 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: listed object 0... 2024-09-15T20:32:52.833 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: listed object 25... 2024-09-15T20:32:52.833 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: listed object 50... 2024-09-15T20:32:52.833 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: listed object 75... 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: listed object 100... 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: listed object 125... 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: saw 150 objects 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_15_[29803]: shutting down. 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_16_[29804]: starting. 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_16_[29804]: added 25 objects... 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_16_[29804]: added half of the objects 2024-09-15T20:32:52.834 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_16_[29804]: added 50 objects... 2024-09-15T20:32:52.835 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_16_[29804]: added 50 objects 2024-09-15T20:32:52.835 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: process_16_[29804]: shutting down. 2024-09-15T20:32:52.835 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.835 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.835 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.835 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.835 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******************************* 2024-09-15T20:32:52.835 INFO:tasks.workunit.client.0.smithi146.stdout: list_parallel: ******* SUCCESS ********** 2024-09-15T20:32:52.875 INFO:tasks.workunit.client.0.smithi146.stdout:leWrite_smithi146-28169-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:52.875 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.836467+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/3886350322' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28339-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:52.875 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.836858+0000 mon.a [INF] from='client.4759 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "SimpleWrite_smithi146-28169-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:52.875 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.837627+0000 mon.a [INF] from='client.4765 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28339-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:52.875 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.843094+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/1609892556' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi146-28177-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.844050+0000 mon.a [INF] from='client.4756 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi146-28177-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.921856+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/856967508' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi146-28423-7"} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.923729+0000 mon.a [INF] from='client.4876 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi146-28423-7"} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.925583+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/856967508' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi146-28423-7"} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.927312+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/3177474870' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi146-28435-7"} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.931801+0000 client.admin [INF] onexx 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.941603+0000 mon.a [INF] from='client.4876 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatEC_smithi146-28423-7"} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.943444+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/856967508' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi146-28423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.945828+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/4285310937' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi146-28193-24"} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.954186+0000 mon.a [INF] from='client.4873 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi146-28435-7"} : dispatch 2024-09-15T20:32:52.876 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.957017+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/3177474870' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi146-28435-7"} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.965232+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3765300722' entity='client.admin' cmd={"prefix":"log", "logtext":["onexx"]} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.965707+0000 mon.a [INF] from='client.4876 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi146-28423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.966034+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoECPP_smithi146-28193-24"} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.966399+0000 mon.a [INF] from='client.4873 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosStatECPP_smithi146-28435-7"} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.969326+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/3177474870' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi146-28435-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.969534+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/4285310937' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi146-28193-24"} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.976772+0000 mon.a [INF] from='client.4873 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi146-28435-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.977227+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoECPP_smithi146-28193-24"} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.977727+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/4285310937' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi146-28193-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:51.978591+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi146-28193-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:52.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.783642+0000 mon.a [INF] from='client.4759 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "SimpleWrite_smithi146-28169-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:52.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.783832+0000 mon.a [INF] from='client.4765 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28339-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:52.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.783993+0000 mon.a [INF] from='client.4756 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi146-28177-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:52.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.784147+0000 mon.a [INF] from='client.4876 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatEC_smithi146-28423-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T20:32:52.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.784305+0000 mon.a [INF] from='client.4873 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosStatECPP_smithi146-28435-7", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T20:32:52.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.784445+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosIoECPP_smithi146-28193-24", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T20:32:53.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.794611+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/856967508' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi146-28423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi146-28423-7"} : dispatch 2024-09-15T20:32:53.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.795298+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/4285310937' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi146-28193-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi146-28193-24"} : dispatch 2024-09-15T20:32:53.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.796635+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/3177474870' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi146-28435-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi146-28435-7"} : dispatch 2024-09-15T20:32:53.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.823488+0000 mon.a [INF] from='client.4876 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatEC_smithi146-28423-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatEC_smithi146-28423-7"} : dispatch 2024-09-15T20:32:53.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.824499+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosIoECPP_smithi146-28193-24", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoECPP_smithi146-28193-24"} : dispatch 2024-09-15T20:32:53.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.825428+0000 mon.a [INF] from='client.4873 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosStatECPP_smithi146-28435-7", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosStatECPP_smithi146-28435-7"} : dispatch 2024-09-15T20:32:53.877 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.837577+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/1732815460' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28320-4-cache","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.844635+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/406220101' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi146-28936-1"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.845659+0000 mon.a [INF] from='client.4678 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi146-28936-1"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.846620+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/3497999582' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28644-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.848253+0000 mon.a [INF] from='client.4894 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28644-2","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.870884+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3765300722' entity='client.admin' cmd='[{"prefix":"log", "logtext":["onexx"]}]': finished 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.901235+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2502594869' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi146-28225-10"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.904980+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2253276610' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi146-28217-10"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.918307+0000 mon.a [INF] from='client.4963 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockECPP_smithi146-28225-10"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.918807+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2502594869' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi146-28225-10"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.919300+0000 mon.a [INF] from='client.4966 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosLockEC_smithi146-28217-10"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.920008+0000 mon.a [INF] from='client.4963 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockECPP_smithi146-28225-10"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.920395+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2253276610' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi146-28217-10"} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.920957+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2502594869' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi146-28225-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:53.878 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.921950+0000 mon.a [INF] from='client.4966 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosLockEC_smithi146-28217-10"} : dispatch 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.922678+0000 mon.a [INF] from='client.4963 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi146-28225-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.923858+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2253276610' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi146-28217-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:52.929380+0000 mon.a [INF] from='client.4966 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi146-28217-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.793176+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosIoEC_smithi146-28185-16", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosIoEC_smithi146-28185-16"}]': finished 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.793468+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/1732815460' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28320-4-cache","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.793731+0000 mon.a [INF] from='client.4678 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-SimpleWritesmithi146-28936-1"}]': finished 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.793958+0000 mon.a [INF] from='client.4894 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28644-2","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.794185+0000 mon.a [INF] from='client.4963 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockECPP_smithi146-28225-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T20:32:53.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.794413+0000 mon.a [INF] from='client.4966 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-LibRadosLockEC_smithi146-28217-10", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd open_pools_parallel: process_1_[28794]: starting. 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_1_[28794]: creating pool ceph_test_rados_open_pools_parallel.smithi146-28731 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_1_[28794]: created object 0... 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_1_[28794]: created object 25... 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_1_[28794]: created object 49... 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_1_[28794]: finishing. 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_1_[28794]: shutting down. 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_2_[28795]: starting. 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_2_[28795]: rados_pool_create. 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_2_[28795]: rados_ioctx_create. 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_2_[28795]: shutting down. 2024-09-15T20:32:54.855 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: ******************************* 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_3_[29495]: starting. 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_3_[29495]: creating pool ceph_test_rados_open_pools_parallel.smithi146-28731 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_3_[29495]: created object 0... 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_3_[29495]: created object 25... 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_3_[29495]: created object 49... 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_3_[29495]: finishing. 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_3_[29495]: shutting down. 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: ******************************* 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_4_[29496]: starting. 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_4_[29496]: rados_pool_create. 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_4_[29496]: rados_ioctx_create. 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: process_4_[29496]: shutting down. 2024-09-15T20:32:54.856 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: ******************************* 2024-09-15T20:32:54.857 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: ******************************* 2024-09-15T20:32:54.857 INFO:tasks.workunit.client.0.smithi146.stdout: open_pools_parallel: ******* SUCCESS ********** 2024-09-15T20:32:54.862 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_1_[28856]: starting. 2024-09-15T20:32:54.862 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_1_[28856]: creating pool ceph_test_rados_delete_pools_parallel.smithi146-28758 2024-09-15T20:32:54.862 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_1_[28856]: created object 0... 2024-09-15T20:32:54.862 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_1_[28856]: created object 25... 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_1_[28856]: created object 49... 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_1_[28856]: finishing. 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_1_[28856]: shutting down. 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_2_[28857]: starting. 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_2_[28857]: deleting pool ceph_test_rados_delete_pools_parallel.smithi146-28758 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_2_[28857]: shutting down. 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: ******************************* 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_3_[29492]: starting. 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_3_[29492]: creating pool ceph_test_rados_delete_pools_parallel.smithi146-28758 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_3_[29492]: created object 0... 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_3_[29492]: created object 25... 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_3_[29492]: created object 49... 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_3_[29492]: finishing. 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_3_[29492]: shutting down. 2024-09-15T20:32:54.863 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: ******************************* 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_5_[29494]: starting. 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_5_[29494]: listing objects. 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_5_[29494]: listed object 0... 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_5_[29494]: listed object 25... 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_5_[29494]: saw 50 objects 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_5_[29494]: shutting down. 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: ******************************* 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_4_[29493]: starting. 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_4_[29493]: deleting pool ceph_test_rados_delete_pools_parallel.smithi146-28758 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: process_4_[29493]: shutting down. 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: ******************************* 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: ******************************* 2024-09-15T20:32:54.864 INFO:tasks.workunit.client.0.smithi146.stdout: delete_pools_parallel: ******* SUCCESS ********** 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: Running main() from gmock_main.cc 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [==========] Running 3 tests from 1 test suite. 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [----------] Global test environment set-up. 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [----------] 3 tests from NeoRadosCmd 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [ RUN ] NeoRadosCmd.MonDescribe 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [ OK ] NeoRadosCmd.MonDescribe (1934 ms) 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [ RUN ] NeoRadosCmd.OSDCmd 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [ OK ] NeoRadosCmd.OSDCmd (2024 ms) 2024-09-15T20:32:55.840 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [ RUN ] NeoRadosCmd.PGCmd 2024-09-15T20:32:55.841 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [ OK ] NeoRadosCmd.PGCmd (3006 ms) 2024-09-15T20:32:55.841 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [----------] 3 tests from NeoRadosCmd (6964 ms total) 2024-09-15T20:32:55.841 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: 2024-09-15T20:32:55.841 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [----------] Global test environment tear-down 2024-09-15T20:32:55.841 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [==========] 3 tests from 1 test suite ran. (6964 ms total) 2024-09-15T20:32:55.841 INFO:tasks.workunit.client.0.smithi146.stdout: cmd: [ PASSED ] 3 tests. 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: Running main() from gmock_main.cc 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [==========] Running 40 tests from 2 test suites. 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [----------] Global test environment set-up. 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: seed 28193 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TooBigPP 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.TooBigPP (0 ms) 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SimpleWritePP 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.SimpleWritePP (224 ms) 2024-09-15T20:32:55.996 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadOpPP 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadOpPP (15 ms) 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadOpPP 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadOpPP (6 ms) 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.SparseReadExtentArrayOpPP 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.SparseReadExtentArrayOpPP (640 ms) 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP (5 ms) 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RoundTripPP2 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.RoundTripPP2 (7 ms) 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.Checksum 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.Checksum (5 ms) 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.ReadIntoBufferlist 2024-09-15T20:32:55.997 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.ReadIntoBufferlist (7 ms) 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.OverlappingWriteRoundTripPP 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.OverlappingWriteRoundTripPP (9 ms) 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP (9 ms) 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.WriteFullRoundTripPP2 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.WriteFullRoundTripPP2 (3 ms) 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.AppendRoundTripPP 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.AppendRoundTripPP (8 ms) 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.TruncTestPP 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.TruncTestPP (5 ms) 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RemoveTestPP 2024-09-15T20:32:55.998 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.RemoveTestPP (6 ms) 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrsRoundTripPP 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrsRoundTripPP (5 ms) 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.RmXattrPP 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.RmXattrPP (17 ms) 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.XattrListPP (13 ms) 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CrcZeroWrite 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.CrcZeroWrite (12 ms) 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtPP 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtPP (228 ms) 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtDNEPP 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtDNEPP (35 ms) 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoPP.CmpExtMismatchPP 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoPP.CmpExtMismatchPP (13 ms) 2024-09-15T20:32:55.999 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [----------] 22 tests from LibRadosIoPP (1273 ms total) 2024-09-15T20:32:56.000 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: 2024-09-15T20:32:56.000 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP 2024-09-15T20:32:56.000 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SimpleWritePP 2024-09-15T20:32:56.000 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SimpleWritePP (1162 ms) 2024-09-15T20:32:56.000 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.ReadOpPP 2024-09-15T20:32:56.000 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.ReadOpPP (15 ms) 2024-09-15T20:32:56.000 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.SparseReadOpPP 2024-09-15T20:32:56.000 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.SparseReadOpPP (3 ms) 2024-09-15T20:32:56.884 INFO:tasks.workunit.client.0.smithi146.stdout: : got: 2024-09-15T20:32:53.798013+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/406220101' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi146-28936-1"} : dispatch 2024-09-15T20:32:56.884 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.798343+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2253276610' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi146-28217-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi146-28217-10"} : dispatch 2024-09-15T20:32:56.884 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.804827+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/2502594869' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi146-28225-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi146-28225-10"} : dispatch 2024-09-15T20:32:56.884 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.831851+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/1609892556' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi146-28177-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:56.884 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.833070+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/649665553' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi146-28502-1"} : dispatch 2024-09-15T20:32:56.884 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.834228+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/1635565085' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi146-29018-1"} : dispatch 2024-09-15T20:32:56.884 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.837660+0000 mon.a [INF] from='client.4678 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"SimpleWritesmithi146-28936-1"} : dispatch 2024-09-15T20:32:56.884 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.838273+0000 mon.a [INF] from='client.4966 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi146-28217-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi146-28217-10"} : dispatch 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.839610+0000 mon.a [INF] from='client.4963 ' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi146-28225-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi146-28225-10"} : dispatch 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.840238+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/1732815460' entity='client.admin' cmd={"prefix": "osd tier add", "pool": "test-rados-api-smithi146-28320-4", "tierpool": "test-rados-api-smithi146-28320-4-cache"} : dispatch 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.842619+0000 mon.a [INF] from='client.4756 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "PoolQuotaPP_smithi146-28177-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.850681+0000 mon.a [INF] from='client.4570 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosWatchNotifyECPP_smithi146-28502-1"} : dispatch 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:53.850912+0000 mon.a [INF] from='client.4684 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-ListObjectssmithi146-29018-1"} : dispatch 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:54.879136+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3765300722' entity='client.admin' cmd='[{"prefix":"log", "logtext":["twoxx"]}]': finished 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:54.913564+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3740492345' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi146-28936-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.809848+0000 mon.a [INF] from='client.4966 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockEC_smithi146-28217-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockEC_smithi146-28217-10"}]': finished 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.810082+0000 mon.a [INF] from='client.4963 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "LibRadosLockECPP_smithi146-28225-10", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-LibRadosLockECPP_smithi146-28225-10"}]': finished 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.810264+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/1732815460' entity='client.admin' cmd='[{"prefix": "osd tier set-overlay", "pool": "test-rados-api-smithi146-28320-4", "overlaypool": "test-rados-api-smithi146-28320-4-cache"}]': finished 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.810413+0000 mon.a [INF] from='client.4570 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"LibRadosWatchNotifyECPP_smithi146-28502-1"}]': finished 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.810532+0000 mon.a [INF] from='client.4684 ' entity='client.admin' cmd='[{"prefix": "osd crush rule rm", "name":"ListObjectssmithi146-29018-1"}]': finished 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.810632+0000 mon.a [INF] from='client.4756 ' entity='client.admin' cmd='[{"prefix": "osd pool set-quota", "pool": "PoolQuotaPP_smithi146-28177-3", "field": "max_bytes", "val": "4096"}]': finished 2024-09-15T20:32:56.885 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.810789+0000 mon.a [INF] from='client.5014 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28339-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:56.886 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.810903+0000 mon.a [INF] from='client.4993 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "WaitForSafe_smithi146-28169-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:56.886 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.811065+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3740492345' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ReadOpsmithi146-28936-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T20:32:56.886 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.820893+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/1732815460' entity='client.admin' cmd={"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi146-28320-4-cache", "mode": "writeback"} : dispatch 2024-09-15T20:32:56.886 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.821864+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3740492345' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ReadOpsmithi146-28936-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ReadOpsmithi146-28936-2"} : dispatch 2024-09-15T20:32:56.886 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.825219+0000 mon.b [INF] from='client.4750 v1:172.21.15.146:0/609894408' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:56.886 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.827831+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/878342955' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28644-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:56.886 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.833572+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.834930+0000 mon.a [INF] from='client.5128 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28644-3","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:55.905490+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3270599773' entity='client.admin' cmd={"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi146-29018-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]} : dispatch 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.053864+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/4285310937' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi146-28193-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.054627+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd={"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi146-28193-24", "var": "allow_ec_overwrites", "val": "true"} : dispatch 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.813264+0000 mon.a [WRN] Health check failed: 1 cache pools are missing hit_sets (CACHE_POOL_NO_HIT_SET) 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.816731+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/1732815460' entity='client.admin' cmd='[{"prefix": "osd tier cache-mode", "pool": "test-rados-api-smithi146-28320-4-cache", "mode": "writeback"}]': finished 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.816855+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosIoEC_smithi146-28185-16"}]': finished 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.817051+0000 mon.a [INF] from='client.5128 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi146-28644-3","app": "rados","yes_i_really_mean_it": true}]': finished 2024-09-15T20:32:56.902 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.817184+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3270599773' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "testprofile-ListObjectsNSsmithi146-29018-2", "profile": [ "k=2", "m=1", "crush-failure-domain=osd"]}]': finished 2024-09-15T20:32:56.903 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.817287+0000 mon.a [INF] from='client.4906 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "LibRadosIoECPP_smithi146-28193-24", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-09-15T20:32:56.903 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.826214+0000 mon.b [INF] from='client.4750 v1:172.21.15.146:0/609894408' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:56.903 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.836924+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3270599773' entity='client.admin' cmd={"prefix": "osd pool create", "pool": "ListObjectsNSsmithi146-29018-2", "pool_type":"erasure", "pg_num":8, "pgp_num":8, "erasure_code_profile":"testprofile-ListObjectsNSsmithi146-29018-2"} : dispatch 2024-09-15T20:32:56.903 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.838556+0000 mon.a [INF] from='client.4750 ' entity='client.admin' cmd={"prefix": "osd crush rule rm", "name":"LibRadosIoEC_smithi146-28185-16"} : dispatch 2024-09-15T20:32:56.903 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.839896+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/3177474870' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi146-28435-7"} : dispatch 2024-09-15T20:32:56.903 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.843988+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/856967508' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi146-28423-7"} : dispatch 2024-09-15T20:32:56.903 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.844979+0000 mon.b [INF] from='client.? v1:172.21.15.146:0/4242486206' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi146-28502-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:32:57.858 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.846176+0000 mon.a [INF] from='client.4873 ' entity='client.admin' cmd={" api_io: Running main() from gmock_main.cc 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [==========] Running 24 tests from 2 test suites. 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [----------] Global test environment set-up. 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [----------] 14 tests from LibRadosIo 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.SimpleWrite 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.SimpleWrite (231 ms) 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.TooBig 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.TooBig (0 ms) 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.ReadTimeout 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: no timeout :/ 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: no timeout :/ 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: no timeout :/ 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: no timeout :/ 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: no timeout :/ 2024-09-15T20:32:57.859 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.ReadTimeout (31 ms) 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.RoundTrip 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.RoundTrip (15 ms) 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.Checksum 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.Checksum (6 ms) 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.OverlappingWriteRoundTrip 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.OverlappingWriteRoundTrip (5 ms) 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.WriteFullRoundTrip 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.WriteFullRoundTrip (3 ms) 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.AppendRoundTrip 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.AppendRoundTrip (8 ms) 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.ZeroLenZero 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.ZeroLenZero (1 ms) 2024-09-15T20:32:57.860 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.TruncTest 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.TruncTest (4 ms) 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.RemoveTest 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.RemoveTest (9 ms) 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.XattrsRoundTrip 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.XattrsRoundTrip (10 ms) 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.RmXattr 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.RmXattr (26 ms) 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIo.XattrIter 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIo.XattrIter (13 ms) 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [----------] 14 tests from LibRadosIo (362 ms total) 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [----------] 10 tests from LibRadosIoEC 2024-09-15T20:32:57.861 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.SimpleWrite 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.SimpleWrite (1175 ms) 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.RoundTrip 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.RoundTrip (22 ms) 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.OverlappingWriteRoundTrip 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.OverlappingWriteRoundTrip (8 ms) 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.WriteFullRoundTrip 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.WriteFullRoundTrip (5 ms) 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.AppendRoundTrip 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.AppendRoundTrip (8 ms) 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.TruncTest 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.TruncTest (9 ms) 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.RemoveTest 2024-09-15T20:32:57.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.RemoveTest (7 ms) 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.XattrsRoundTrip 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.XattrsRoundTrip (5 ms) 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.RmXattr 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.RmXattr (24 ms) 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ RUN ] LibRadosIoEC.XattrIter 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ OK ] LibRadosIoEC.XattrIter (14 ms) 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [----------] 10 tests from LibRadosIoEC (1277 ms total) 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [----------] Global test environment tear-down 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [==========] 24 tests from 2 test suites ran. (9267 ms total) 2024-09-15T20:32:57.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_io: [ PASSED ] 24 tests. 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: Running main() from gmock_main.cc 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [==========] Running 9 tests from 2 test suites. 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [----------] Global test environment set-up. 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [----------] 5 tests from LibRadosStat 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStat.Stat 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStat.Stat (204 ms) 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStat.Stat2 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStat.Stat2 (32 ms) 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStat.StatNS 2024-09-15T20:32:58.862 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStat.StatNS (25 ms) 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStat.ClusterStat 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStat.ClusterStat (0 ms) 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStat.PoolStat 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStat.PoolStat (11 ms) 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [----------] 5 tests from LibRadosStat (272 ms total) 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [----------] 4 tests from LibRadosStatEC 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStatEC.Stat 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStatEC.Stat (1151 ms) 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStatEC.StatNS 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStatEC.StatNS (15 ms) 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStatEC.ClusterStat 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStatEC.ClusterStat (1 ms) 2024-09-15T20:32:58.863 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ RUN ] LibRadosStatEC.PoolStat 2024-09-15T20:32:58.864 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ OK ] LibRadosStatEC.PoolStat (1 ms) 2024-09-15T20:32:58.864 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [----------] 4 tests from LibRadosStatEC (1168 ms total) 2024-09-15T20:32:58.864 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: 2024-09-15T20:32:58.864 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [----------] Global test environment tear-down 2024-09-15T20:32:58.864 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [==========] 9 tests from 2 test suites ran. (10169 ms total) 2024-09-15T20:32:58.864 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat: [ PASSED ] 9 tests. 2024-09-15T20:32:58.866 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: Running main() from gmock_main.cc 2024-09-15T20:32:58.866 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [==========] Running 2 tests from 1 test suite. 2024-09-15T20:32:58.866 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [----------] Global test environment set-up. 2024-09-15T20:32:58.866 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify 2024-09-15T20:32:58.866 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotify 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: handle_notify cookie 94096614599968 notify_id 98784247809 notifier_gid 4720 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotify (2838 ms) 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [ RUN ] NeoRadosWatchNotify.WatchNotifyTimeout 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: Trying... 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: handle_notify cookie 94096626646336 notify_id 111669149700 notifier_gid 4879 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: Waiting for 3.000000000s 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: Timed out. 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: Flushing... 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: Flushed... 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [ OK ] NeoRadosWatchNotify.WatchNotifyTimeout (7027 ms) 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [----------] 2 tests from NeoRadosWatchNotify (9865 ms total) 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [----------] Global test environment tear-down 2024-09-15T20:32:58.867 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [==========] 2 tests from 1 test suite ran. (9865 ms total) 2024-09-15T20:32:58.868 INFO:tasks.workunit.client.0.smithi146.stdout: watch_notify: [ PASSED ] 2 tests. 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: Running main() from gmock_main.cc 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [==========] Running 9 tests from 2 test suites. 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [----------] Global test environment set-up. 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: seed 28435 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPP 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPP (219 ms) 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.Stat2Mtime2PP 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatPP.Stat2Mtime2PP (24 ms) 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.ClusterStatPP 2024-09-15T20:32:58.871 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatPP.ClusterStatPP (0 ms) 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.PoolStatPP 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatPP.PoolStatPP (7 ms) 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatPP.StatPPNS 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatPP.StatPPNS (30 ms) 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [----------] 5 tests from LibRadosStatPP (280 ms total) 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPP 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPP (1152 ms) 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.ClusterStatPP 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.ClusterStatPP (1 ms) 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.PoolStatPP 2024-09-15T20:32:58.872 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.PoolStatPP (5 ms) 2024-09-15T20:32:58.873 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ RUN ] LibRadosStatECPP.StatPPNS 2024-09-15T20:32:58.873 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ OK ] LibRadosStatECPP.StatPPNS (16 ms) 2024-09-15T20:32:58.873 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [----------] 4 tests from LibRadosStatECPP (1174 ms total) 2024-09-15T20:32:58.873 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: 2024-09-15T20:32:58.873 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [----------] Global test environment tear-down 2024-09-15T20:32:58.873 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [==========] 9 tests from 2 test suites ran. (10165 ms total) 2024-09-15T20:32:58.873 INFO:tasks.workunit.client.0.smithi146.stdout: api_stat_pp: [ PASSED ] 9 tests. 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: Running main() from gmock_main.cc 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [==========] Running 3 tests from 1 test suite. 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [----------] Global test environment set-up. 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [----------] 3 tests from NeoradosList 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [ RUN ] NeoradosList.ListObjects 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [ OK ] NeoradosList.ListObjects (2864 ms) 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [ RUN ] NeoradosList.ListObjectsNS 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [ OK ] NeoradosList.ListObjectsNS (3040 ms) 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [ RUN ] NeoradosList.ListObjectsMany 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [ OK ] NeoradosList.ListObjectsMany (4039 ms) 2024-09-15T20:32:58.876 INFO:tasks.workunit.client.0.smithi146.stdout: list: [----------] 3 tests from NeoradosList (9943 ms total) 2024-09-15T20:32:58.877 INFO:tasks.workunit.client.0.smithi146.stdout: list: 2024-09-15T20:32:58.877 INFO:tasks.workunit.client.0.smithi146.stdout: list: [----------] Global test environment tear-down 2024-09-15T20:32:58.877 INFO:tasks.workunit.client.0.smithi146.stdout: list: [==========] 3 tests from 1 test suite ran. (9943 ms total) 2024-09-15T20:32:58.877 INFO:tasks.workunit.client.0.smithi146.stdout: list: [ PASSED ] 3 tests. 2024-09-15T20:33:00.985 INFO:tasks.workunit.client.0.smithi146.stdout:prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatECPP_smithi146-28435-7"} : dispatch 2024-09-15T20:33:00.985 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.869108+0000 mon.a [INF] from='client.4876 ' entity='client.admin' cmd={"prefix": "osd erasure-code-profile rm", "name": "testprofile-LibRadosStatEC_smithi146-28423-7"} : dispatch 2024-09-15T20:33:00.985 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.869525+0000 mon.a [INF] from='client.5164 ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP_smithi146-28502-4","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-09-15T20:33:00.985 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.882288+0000 client.admin [INF] threexx 2024-09-15T20:33:00.985 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: got: 2024-09-15T20:32:56.882766+0000 mon.a [INF] from='client.? v1:172.21.15.146:0/3765300722' entity='client.admin' cmd={"prefix":"log", "logtext":["threexx"]} : dispatch 2024-09-15T20:33:00.985 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ OK ] LibRadosCmd.WatchLog (9177 ms) 2024-09-15T20:33:00.985 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [----------] 4 tests from LibRadosCmd (12241 ms total) 2024-09-15T20:33:00.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: 2024-09-15T20:33:00.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [----------] Global test environment tear-down 2024-09-15T20:33:00.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [==========] 4 tests from 1 test suite ran. (12241 ms total) 2024-09-15T20:33:00.986 INFO:tasks.workunit.client.0.smithi146.stdout: api_cmd: [ PASSED ] 4 tests. 2024-09-15T20:33:01.859 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: Running main() from gmock_main.cc 2024-09-15T20:33:01.859 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [==========] Running 2 tests from 1 test suite. 2024-09-15T20:33:01.859 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [----------] Global test environment set-up. 2024-09-15T20:33:01.859 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [----------] 2 tests from NeoRadosECIo 2024-09-15T20:33:01.859 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [ RUN ] NeoRadosECIo.SimpleWrite 2024-09-15T20:33:01.859 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [ OK ] NeoRadosECIo.SimpleWrite (5882 ms) 2024-09-15T20:33:01.860 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [ RUN ] NeoRadosECIo.ReadOp 2024-09-15T20:33:01.860 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [ OK ] NeoRadosECIo.ReadOp (7039 ms) 2024-09-15T20:33:01.860 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [----------] 2 tests from NeoRadosECIo (12921 ms total) 2024-09-15T20:33:01.860 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: 2024-09-15T20:33:01.860 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [----------] Global test environment tear-down 2024-09-15T20:33:01.860 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [==========] 2 tests from 1 test suite ran. (12921 ms total) 2024-09-15T20:33:01.860 INFO:tasks.workunit.client.0.smithi146.stdout: ec_io: [ PASSED ] 2 tests. 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: Running main() from gmock_main.cc 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [==========] Running 16 tests from 2 test suites. 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [----------] Global test environment set-up. 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: seed 28225 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusivePP 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusivePP (166 ms) 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedPP 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedPP (69 ms) 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockExclusiveDurPP 2024-09-15T20:33:01.879 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockExclusiveDurPP (1108 ms) 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockSharedDurPP 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockSharedDurPP (1013 ms) 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.LockMayRenewPP 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockPP.LockMayRenewPP (4 ms) 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.UnlockPP 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockPP.UnlockPP (5 ms) 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.ListLockersPP 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockPP.ListLockersPP (15 ms) 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockPP.BreakLockPP 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockPP.BreakLockPP (4 ms) 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockPP (2385 ms total) 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: 2024-09-15T20:33:01.880 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusivePP 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusivePP (1147 ms) 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedPP 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedPP (9 ms) 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockExclusiveDurPP 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockExclusiveDurPP (1081 ms) 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockSharedDurPP 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockSharedDurPP (1029 ms) 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.LockMayRenewPP 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.LockMayRenewPP (9 ms) 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.UnlockPP 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.UnlockPP (12 ms) 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.ListLockersPP 2024-09-15T20:33:01.881 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.ListLockersPP (12 ms) 2024-09-15T20:33:01.882 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ RUN ] LibRadosLockECPP.BreakLockPP 2024-09-15T20:33:01.882 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ OK ] LibRadosLockECPP.BreakLockPP (8 ms) 2024-09-15T20:33:01.882 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [----------] 8 tests from LibRadosLockECPP (3307 ms total) 2024-09-15T20:33:01.882 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: 2024-09-15T20:33:01.882 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [----------] Global test environment tear-down 2024-09-15T20:33:01.882 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [==========] 16 tests from 2 test suites ran. (13253 ms total) 2024-09-15T20:33:01.882 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock_pp: [ PASSED ] 16 tests. 2024-09-15T20:33:01.897 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: Running main() from gmock_main.cc 2024-09-15T20:33:01.897 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [==========] Running 16 tests from 2 test suites. 2024-09-15T20:33:01.897 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [----------] Global test environment set-up. 2024-09-15T20:33:01.897 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [----------] 8 tests from LibRadosLock 2024-09-15T20:33:01.897 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusive 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLock.LockExclusive (192 ms) 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLock.LockShared 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLock.LockShared (19 ms) 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLock.LockExclusiveDur 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLock.LockExclusiveDur (1103 ms) 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLock.LockSharedDur 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLock.LockSharedDur (1014 ms) 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLock.LockMayRenew 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLock.LockMayRenew (6 ms) 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLock.Unlock 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLock.Unlock (4 ms) 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLock.ListLockers 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLock.ListLockers (11 ms) 2024-09-15T20:33:01.898 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLock.BreakLock 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLock.BreakLock (5 ms) 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [----------] 8 tests from LibRadosLock (2354 ms total) 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [----------] 8 tests from LibRadosLockEC 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusive 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusive (1128 ms) 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLockEC.LockShared 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLockEC.LockShared (26 ms) 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLockEC.LockExclusiveDur 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLockEC.LockExclusiveDur (1072 ms) 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLockEC.LockSharedDur 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLockEC.LockSharedDur (1018 ms) 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLockEC.LockMayRenew 2024-09-15T20:33:01.899 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLockEC.LockMayRenew (8 ms) 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLockEC.Unlock 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLockEC.Unlock (11 ms) 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLockEC.ListLockers 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLockEC.ListLockers (12 ms) 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ RUN ] LibRadosLockEC.BreakLock 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ OK ] LibRadosLockEC.BreakLock (9 ms) 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [----------] 8 tests from LibRadosLockEC (3284 ms total) 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [----------] Global test environment tear-down 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [==========] 16 tests from 2 test suites ran. (13283 ms total) 2024-09-15T20:33:01.900 INFO:tasks.workunit.client.0.smithi146.stdout: api_lock: [ PASSED ] 16 tests. 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: Running main() from gmock_main.cc 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [==========] Running 6 tests from 1 test suite. 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [----------] Global test environment set-up. 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [----------] 6 tests from NeoRadosPools 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ RUN ] NeoRadosPools.PoolList 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ OK ] NeoRadosPools.PoolList (1845 ms) 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ RUN ] NeoRadosPools.PoolLookup 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ OK ] NeoRadosPools.PoolLookup (2027 ms) 2024-09-15T20:33:02.503 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ RUN ] NeoRadosPools.PoolLookupOtherInstance 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ OK ] NeoRadosPools.PoolLookupOtherInstance (2017 ms) 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ RUN ] NeoRadosPools.PoolDelete 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ OK ] NeoRadosPools.PoolDelete (4011 ms) 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateDelete 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ OK ] NeoRadosPools.PoolCreateDelete (2016 ms) 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ RUN ] NeoRadosPools.PoolCreateWithCrushRule 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ OK ] NeoRadosPools.PoolCreateWithCrushRule (1623 ms) 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [----------] 6 tests from NeoRadosPools (13539 ms total) 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [----------] Global test environment tear-down 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [==========] 6 tests from 1 test suite ran. (13539 ms total) 2024-09-15T20:33:02.504 INFO:tasks.workunit.client.0.smithi146.stdout: pool: [ PASSED ] 6 tests. 2024-09-15T20:33:05.024 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: Running main() from gmock_main.cc 2024-09-15T20:33:05.024 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [==========] Running 16 tests from 2 test suites. 2024-09-15T20:33:05.024 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [----------] Global test environment set-up. 2024-09-15T20:33:05.024 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP 2024-09-15T20:33:05.024 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotify 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: notify 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotify (1137 ms) 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyECPP.WatchNotifyTimeout 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyECPP.WatchNotifyTimeout (16 ms) 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [----------] 2 tests from LibRadosWatchNotifyECPP (1154 ms total) 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: notify 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/0 (122 ms) 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: notify 2024-09-15T20:33:05.025 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify/1 (3991 ms) 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/0 (10 ms) 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotifyTimeout/1 (10 ms) 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703332958672 notify_id 146028888067 notifier_gid 5164 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/0 (6 ms) 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703332961984 notify_id 146028888070 notifier_gid 5164 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2/1 (6 ms) 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703332961984 notify_id 146028888071 notifier_gid 5164 2024-09-15T20:33:05.026 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/0 (6 ms) 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703332961984 notify_id 146028888072 notifier_gid 5164 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioWatchNotify2/1 (7 ms) 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703332958672 notify_id 146028888073 notifier_gid 5164 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/0 (10 ms) 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703331684688 notify_id 146028888068 notifier_gid 5164 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.AioNotify/1 (6 ms) 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: trying... 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703331684688 notify_id 146028888074 notifier_gid 5164 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: timed out 2024-09-15T20:33:05.027 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: flushing 2024-09-15T20:33:06.574 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_n api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP 2024-09-15T20:33:06.574 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP (3 ms) 2024-09-15T20:33:06.574 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RoundTripPP2 2024-09-15T20:33:06.574 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RoundTripPP2 (3 ms) 2024-09-15T20:33:06.574 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.OverlappingWriteRoundTripPP 2024-09-15T20:33:06.574 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.OverlappingWriteRoundTripPP (12 ms) 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP (5 ms) 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.WriteFullRoundTripPP2 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.WriteFullRoundTripPP2 (5 ms) 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.AppendRoundTripPP 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.AppendRoundTripPP (7 ms) 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.TruncTestPP 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.TruncTestPP (4 ms) 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RemoveTestPP 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RemoveTestPP (4 ms) 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrsRoundTripPP 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrsRoundTripPP (4 ms) 2024-09-15T20:33:06.575 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.RmXattrPP 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.RmXattrPP (12 ms) 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CrcZeroWrite 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CrcZeroWrite (6442 ms) 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.XattrListPP 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.XattrListPP (1138 ms) 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtPP 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtPP (10 ms) 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtDNEPP 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtDNEPP (4 ms) 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ RUN ] LibRadosIoECPP.CmpExtMismatchPP 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ OK ] LibRadosIoECPP.CmpExtMismatchPP (6 ms) 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [----------] 18 tests from LibRadosIoECPP (8839 ms total) 2024-09-15T20:33:06.576 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: 2024-09-15T20:33:06.577 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [----------] Global test environment tear-down 2024-09-15T20:33:06.577 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [==========] 40 tests from 2 test suites ran. (17975 ms total) 2024-09-15T20:33:06.577 INFO:tasks.workunit.client.0.smithi146.stdout: api_io_pp: [ PASSED ] 40 tests. 2024-09-15T20:33:07.554 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: Running main() from gmock_main.cc 2024-09-15T20:33:07.554 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [==========] Running 11 tests from 2 test suites. 2024-09-15T20:33:07.554 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [----------] Global test environment set-up. 2024-09-15T20:33:07.554 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify 2024-09-15T20:33:07.554 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify 2024-09-15T20:33:07.554 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify_test_cb 2024-09-15T20:33:07.554 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify (267 ms) 2024-09-15T20:33:07.554 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch2Delete 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94310613717824 err -107 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: waiting up to 300 for disconnect notification ... 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch2Delete (15 ms) 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: waiting up to watch_notify2_test_errcb300 cookie 94310613733296 err -107 for disconnect notification ... 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete (14 ms) 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 98784247810 cookie 94310613733296 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2 (17 ms) 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchNotify2 2024-09-15T20:33:07.555 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 98784247808 cookie 94310613777424 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchNotify2 (16 ms) 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioNotify 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 98784247811 cookie 94310613777344 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioNotify (7 ms) 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Multi 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 98784247809 cookie 94310613777344 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 98784247809 cookie 94310613780736 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Multi (20 ms) 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.WatchNotify2Timeout 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 98784247808 cookie 94310613780736 2024-09-15T20:33:07.556 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 98784247809 cookie 94310613780736 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 103079215106 cookie 94310613780736 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.WatchNotify2Timeout (3013 ms) 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.Watch3Timeout 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: waiting up to 1024 for osd to time us out ... 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94310613780736 err -107 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_cb from 4567 notify_id 133143986180 cookie 94310613780736 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.Watch3Timeout (5016 ms) 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotify.AioWatchDelete2 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: waiting up to 30 for disconnect notification ... 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify2_test_errcb cookie 94310613780736 err -107 2024-09-15T20:33:07.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotify.AioWatchDelete2 (1013 ms) 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [----------] 10 tests from LibRadosWatchNotify (9398 ms total) 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ RUN ] LibRadosWatchNotifyEC.WatchNotify 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: watch_notify_test_cb 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ OK ] LibRadosWatchNotifyEC.WatchNotify (1121 ms) 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [----------] 1 test from LibRadosWatchNotifyEC (1121 ms total) 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: 2024-09-15T20:33:07.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [----------] Global test environment tear-down 2024-09-15T20:33:07.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [==========] 11 tests from 2 test suites ran. (18846 ms total) 2024-09-15T20:33:07.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify: [ PASSED ] 11 tests. 2024-09-15T20:33:12.517 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: Running main() from gmock_main.cc 2024-09-15T20:33:12.517 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [==========] Running 3 tests from 1 test suite. 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [----------] Global test environment set-up. 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [----------] 3 tests from NeoradosECList 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [ RUN ] NeoradosECList.ListObjects 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [ OK ] NeoradosECList.ListObjects (6887 ms) 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsNS 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsNS (6653 ms) 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [ RUN ] NeoradosECList.ListObjectsMany 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [ OK ] NeoradosECList.ListObjectsMany (10027 ms) 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [----------] 3 tests from NeoradosECList (23567 ms total) 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [----------] Global test environment tear-down 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [==========] 3 tests from 1 test suite ran. (23567 ms total) 2024-09-15T20:33:12.518 INFO:tasks.workunit.client.0.smithi146.stdout: ec_list: [ PASSED ] 3 tests. 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: Running main() from gmock_main.cc 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [==========] Running 8 tests from 2 test suites. 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [----------] Global test environment set-up. 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ RUN ] LibradosCWriteOps.NewDelete 2024-09-15T20:33:14.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ OK ] LibradosCWriteOps.NewDelete (0 ms) 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [----------] 1 test from LibradosCWriteOps (0 ms total) 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.assertExists 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.assertExists (3028 ms) 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteOpAssertVersion 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteOpAssertVersion (3052 ms) 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Xattrs 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Xattrs (3003 ms) 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Write 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Write (3022 ms) 2024-09-15T20:33:14.532 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.Exec 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.Exec (2646 ms) 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.WriteSame 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.WriteSame (3024 ms) 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ RUN ] LibRadosCWriteOps.CmpExt 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ OK ] LibRadosCWriteOps.CmpExt (7977 ms) 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [----------] 7 tests from LibRadosCWriteOps (25752 ms total) 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [----------] Global test environment tear-down 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [==========] 8 tests from 2 test suites ran. (25752 ms total) 2024-09-15T20:33:14.533 INFO:tasks.workunit.client.0.smithi146.stdout: api_c_write_operations: [ PASSED ] 8 tests. 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: Running main() from gmock_main.cc 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [==========] Running 7 tests from 1 test suite. 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [----------] Global test environment set-up. 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertExists 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertExists (2835 ms) 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ RUN ] NeoRadosWriteOps.AssertVersion 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ OK ] NeoRadosWriteOps.AssertVersion (3035 ms) 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Xattrs 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ OK ] NeoRadosWriteOps.Xattrs (3005 ms) 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Write 2024-09-15T20:33:14.534 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ OK ] NeoRadosWriteOps.Write (3008 ms) 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ RUN ] NeoRadosWriteOps.Exec 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ OK ] NeoRadosWriteOps.Exec (2658 ms) 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ RUN ] NeoRadosWriteOps.WriteSame 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ OK ] NeoRadosWriteOps.WriteSame (3017 ms) 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ RUN ] NeoRadosWriteOps.CmpExt 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ OK ] NeoRadosWriteOps.CmpExt (7983 ms) 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [----------] 7 tests from NeoRadosWriteOps (25541 ms total) 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [----------] Global test environment tear-down 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [==========] 7 tests from 1 test suite ran. (25541 ms total) 2024-09-15T20:33:14.535 INFO:tasks.workunit.client.0.smithi146.stdout: write_operations: [ PASSED ] 7 tests. 2024-09-15T20:33:16.346 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [==========] Running 4 tests from 1 test suite. 2024-09-15T20:33:16.346 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [----------] Global test environment set-up. 2024-09-15T20:33:16.346 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [----------] 4 tests from LibRadosService 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ RUN ] LibRadosService.RegisterEarly 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ OK ] LibRadosService.RegisterEarly (5053 ms) 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ RUN ] LibRadosService.RegisterLate 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ OK ] LibRadosService.RegisterLate (131 ms) 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ RUN ] LibRadosService.StatusFormat 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: cluster: 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: id: a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: health: HEALTH_WARN 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 12 pool(s) do not have an application enabled 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: services: 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: mon: 3 daemons, quorum a,b (age 45s), out of quorum: c 2024-09-15T20:33:16.347 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: mgr: x(active, since 81s) 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: osd: 6 osds: 6 up (since 76s), 6 in (since 84s) 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: laundry: 2 daemons active (1 hosts) 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: data: 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: pools: 23 pools, 609 pgs 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: objects: 31 objects, 579 KiB 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: usage: 160 MiB used, 540 GiB / 540 GiB avail 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: pgs: 88.506% pgs unknown 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 0.328% pgs not active 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 539 unknown 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 68 active+clean 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2 creating+peering 2024-09-15T20:33:16.348 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: io: 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: client: 511 B/s rd, 13 KiB/s wr, 0 op/s rd, 19 op/s wr 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: cluster: 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: id: a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: health: HEALTH_WARN 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 14 pool(s) do not have an application enabled 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: services: 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: mon: 3 daemons, quorum a,b (age 47s), out of quorum: c 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: mgr: x(active, since 83s) 2024-09-15T20:33:16.349 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: osd: 6 osds: 6 up (since 78s), 6 in (since 86s) 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: foo: 16 portals active (1 hosts, 3 zones) 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: laundry: 1 daemon active (1 hosts) 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: data: 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: pools: 26 pools, 753 pgs 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: objects: 51 objects, 581 KiB 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: usage: 182 MiB used, 540 GiB / 540 GiB avail 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: pgs: 35.989% pgs unknown 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 25.896% pgs not active 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 287 active+clean 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 271 unknown 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 195 creating+peering 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.350 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: io: 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: client: 1.7 KiB/s rd, 23 KiB/s wr, 179 op/s rd, 213 op/s wr 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ OK ] LibRadosService.StatusFormat (2314 ms) 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ RUN ] LibRadosService.Status 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ OK ] LibRadosService.Status (20044 ms) 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [----------] 4 tests from LibRadosService (27542 ms total) 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [----------] Global test environment tear-down 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [==========] 4 tests from 1 test suite ran. (27543 ms total) 2024-09-15T20:33:16.351 INFO:tasks.workunit.client.0.smithi146.stdout: api_service: [ PASSED ] 4 tests. 2024-09-15T20:33:17.550 INFO:tasks.workunit.client.0.smithi146.stdout:otify_pp: flushed 2024-09-15T20:33:17.550 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/0 (3004 ms) 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: trying... 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703331684688 notify_id 158913789957 notifier_gid 5164 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: timed out 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: flushing 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: flushed 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify2Timeout/1 (3012 ms) 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: List watches 2024-09-15T20:33:17.551 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: notify2 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703331684688 notify_id 171798691851 notifier_gid 5164 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: notify2 done 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: watch_check 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: unwatch2 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: flushing 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: done 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/0 (3010 ms) 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ RUN ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: List watches 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: notify2 2024-09-15T20:33:17.552 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: handle_notify cookie 94703331684688 notify_id 184683593740 notifier_gid 5164 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: notify2 done 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: watch_check 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: unwatch2 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: flushing 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: done 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ OK ] LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP.WatchNotify3/1 (3018 ms) 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [----------] 14 tests from LibRadosWatchNotifyPPTests/LibRadosWatchNotifyPP (16218 ms total) 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [----------] Global test environment tear-down 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [==========] 16 tests from 2 test suites ran. (28829 ms total) 2024-09-15T20:33:17.553 INFO:tasks.workunit.client.0.smithi146.stdout: api_watch_notify_pp: [ PASSED ] 16 tests. 2024-09-15T20:33:21.560 INFO:tasks.workunit.client.0.smithi146.stdout: misc: Running main() from gmock_main.cc 2024-09-15T20:33:21.560 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [==========] Running 12 tests from 1 test suite. 2024-09-15T20:33:21.560 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [----------] Global test environment set-up. 2024-09-15T20:33:21.560 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [----------] 12 tests from NeoRadosMisc 2024-09-15T20:33:21.560 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.Version 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.Version (1852 ms) 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.WaitOSDMap 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.WaitOSDMap (2020 ms) 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.LongName 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.LongName (3003 ms) 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.LongLocator 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.LongLocator (3025 ms) 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.LongNamespace 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.LongNamespace (3034 ms) 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.LongAttrName 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.LongAttrName (2630 ms) 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.Exec 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.Exec (3018 ms) 2024-09-15T20:33:21.561 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.Operate1 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.Operate1 (3024 ms) 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.Operate2 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.Operate2 (2969 ms) 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.BigObject 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.BigObject (3032 ms) 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.BigAttr 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.BigAttr (1945 ms) 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ RUN ] NeoRadosMisc.WriteSame 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ OK ] NeoRadosMisc.WriteSame (3048 ms) 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [----------] 12 tests from NeoRadosMisc (32601 ms total) 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [----------] Global test environment tear-down 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [==========] 12 tests from 1 test suite ran. (32601 ms total) 2024-09-15T20:33:21.562 INFO:tasks.workunit.client.0.smithi146.stdout: misc: [ PASSED ] 12 tests. 2024-09-15T20:33:24.358 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [==========] Running 4 tests from 1 test suite. 2024-09-15T20:33:24.358 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [----------] Global test environment set-up. 2024-09-15T20:33:24.358 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP 2024-09-15T20:33:24.358 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterEarly 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterEarly (5066 ms) 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ RUN ] LibRadosServicePP.RegisterLate 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ OK ] LibRadosServicePP.RegisterLate (129 ms) 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Status 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ OK ] LibRadosServicePP.Status (20058 ms) 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ RUN ] LibRadosServicePP.Close 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: attempt 0 of 20 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: failed to deregister: 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: { 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "epoch": 9, 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "modified": "2024-09-15T20:33:16.683438+0000", 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "services": { 2024-09-15T20:33:24.359 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "laundry": { 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "daemons": { 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "summary": "", 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "close-test-pid28633": { 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "start_epoch": 8, 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "start_stamp": "2024-09-15T20:33:14.092282+0000", 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "gid": 11044, 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "addr": "172.21.15.146:0/2062331052", 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "metadata": { 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "arch": "x86_64", 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "ceph_release": "squid", 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "ceph_version": "ceph version 19.3.0-4930-gfc3e0bff (fc3e0bff7468fd55478adbae7f75508b749124b6) squid (dev)", 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "ceph_version_short": "19.3.0-4930-gfc3e0bff", 2024-09-15T20:33:24.360 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "cpu": "Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "distro": "ubuntu", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "distro_description": "Ubuntu 22.04.1 LTS", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "distro_version": "22.04", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "foo": "bar", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "hostname": "smithi146", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "kernel_description": "#131-Ubuntu SMP Fri Aug 9 08:29:53 UTC 2024", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "kernel_version": "5.15.0-121-generic", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "mem_swap_kb": "2097148", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "mem_total_kb": "32746832", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "os": "Linux", 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "this": "that" 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: }, 2024-09-15T20:33:24.361 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: "task_status": {} 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: } 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: } 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: } 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: } 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: } 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: attempt 1 of 20 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ OK ] LibRadosServicePP.Close (10325 ms) 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [----------] 4 tests from LibRadosServicePP (35578 ms total) 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [----------] Global test environment tear-down 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [==========] 4 tests from 1 test suite ran. (35578 ms total) 2024-09-15T20:33:24.362 INFO:tasks.workunit.client.0.smithi146.stdout: api_service_pp: [ PASSED ] 4 tests. 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: Running main() from gmock_main.cc 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [==========] Running 9 tests from 1 test suite. 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [----------] Global test environment set-up. 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [----------] 9 tests from LibRadosPools 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolList 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolList (2165 ms) 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup (3018 ms) 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookup2 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolLookup2 (3018 ms) 2024-09-15T20:33:25.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolLookupOtherInstance 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolLookupOtherInstance (3025 ms) 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolReverseLookupOtherInstance 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolReverseLookupOtherInstance (2640 ms) 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolDelete 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolDelete (5016 ms) 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateDelete 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateDelete (4984 ms) 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolCreateWithCrushRule 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolCreateWithCrushRule (4980 ms) 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ RUN ] LibRadosPools.PoolGetBaseTier 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ OK ] LibRadosPools.PoolGetBaseTier (8064 ms) 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [----------] 9 tests from LibRadosPools (36911 ms total) 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [----------] Global test environment tear-down 2024-09-15T20:33:25.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [==========] 9 tests from 1 test suite ran. (36911 ms total) 2024-09-15T20:33:25.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_pool: [ PASSED ] 9 tests. 2024-09-15T20:33:30.551 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: Running main() from gmock_main.cc 2024-09-15T20:33:30.551 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [==========] Running 14 tests from 1 test suite. 2024-09-15T20:33:30.551 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [----------] Global test environment set-up. 2024-09-15T20:33:30.551 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps 2024-09-15T20:33:30.551 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.SetOpFlags 2024-09-15T20:33:30.551 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.SetOpFlags (2855 ms) 2024-09-15T20:33:30.551 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertExists 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertExists (3027 ms) 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.AssertVersion 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.AssertVersion (3011 ms) 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpXattr 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpXattr (3018 ms) 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.Read 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.Read (2639 ms) 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.Checksum 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.Checksum (3006 ms) 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.RWOrderedRead 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.RWOrderedRead (3038 ms) 2024-09-15T20:33:30.552 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.ShortRead 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.ShortRead (2930 ms) 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.Exec 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.Exec (3052 ms) 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.Stat 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.Stat (2978 ms) 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.Omap 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.Omap (3035 ms) 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.OmapNuls 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.OmapNuls (2962 ms) 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.GetXattrs 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.GetXattrs (2983 ms) 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ RUN ] NeoRadosReadOps.CmpExt 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ OK ] NeoRadosReadOps.CmpExt (3038 ms) 2024-09-15T20:33:30.553 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [----------] 14 tests from NeoRadosReadOps (41573 ms total) 2024-09-15T20:33:30.554 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: 2024-09-15T20:33:30.554 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [----------] Global test environment tear-down 2024-09-15T20:33:30.554 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [==========] 14 tests from 1 test suite ran. (41573 ms total) 2024-09-15T20:33:30.554 INFO:tasks.workunit.client.0.smithi146.stdout: read_operations: [ PASSED ] 14 tests. 2024-09-15T20:33:30.555 INFO:tasks.workunit.client.0.smithi146.stdout: io: Running main() from gmock_main.cc 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [==========] Running 14 tests from 1 test suite. 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [----------] Global test environment set-up. 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [----------] 14 tests from NeoRadosIo 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.Limits 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.Limits (2881 ms) 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.SimpleWrite 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.SimpleWrite (3036 ms) 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.ReadOp 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.ReadOp (3015 ms) 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.SparseRead 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.SparseRead (3016 ms) 2024-09-15T20:33:30.556 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.RoundTrip 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.RoundTrip (2655 ms) 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.ReadIntoBuufferlist 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.ReadIntoBuufferlist (3021 ms) 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.OverlappingWriteRoundTrip 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.OverlappingWriteRoundTrip (3029 ms) 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.WriteFullRoundTrip 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.WriteFullRoundTrip (2922 ms) 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.AppendRoundTrip 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.AppendRoundTrip (3076 ms) 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.Trunc 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.Trunc (2936 ms) 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.Remove 2024-09-15T20:33:30.557 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.Remove (3046 ms) 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.XattrsRoundTrip 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.XattrsRoundTrip (2975 ms) 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.RmXattr 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.RmXattr (3014 ms) 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ RUN ] NeoRadosIo.GetXattrs 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ OK ] NeoRadosIo.GetXattrs (2987 ms) 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [----------] 14 tests from NeoRadosIo (41609 ms total) 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [----------] Global test environment tear-down 2024-09-15T20:33:30.558 INFO:tasks.workunit.client.0.smithi146.stdout: io: [==========] 14 tests from 1 test suite ran. (41609 ms total) 2024-09-15T20:33:30.559 INFO:tasks.workunit.client.0.smithi146.stdout: io: [ PASSED ] 14 tests. 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [==========] Running 12 tests from 4 test suites. 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] Global test environment set-up. 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMiscVersion.Version 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMiscVersion.Version (0 ms) 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure 2024-09-15T20:33:30.567 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectFailure 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: unable to get monitor info from DNS SRV with service name: ceph-mon 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: 2024-09-15T20:32:48.705+0000 7fcd8c022880 -1 failed for service _ceph-mon._tcp 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: 2024-09-15T20:32:48.705+0000 7fcd8c022880 -1 monclient: get_monmap_and_config cannot identify monitors to contact 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectFailure (58 ms) 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMiscConnectFailure.ConnectTimeout 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMiscConnectFailure.ConnectTimeout (5013 ms) 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] 2 tests from LibRadosMiscConnectFailure (5072 ms total) 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] 1 test from LibRadosMiscPool 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMiscPool.PoolCreationRace 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: started 0x556477828df0 2024-09-15T20:33:30.568 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: started 0x556477a57a70 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: started 2 aios 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: waiting 0x556477828df0 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: waiting 0x556477a57a70 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: done. 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMiscPool.PoolCreationRace (5142 ms) 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] 1 test from LibRadosMiscPool (5142 ms total) 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] 8 tests from LibRadosMisc 2024-09-15T20:33:30.569 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMisc.ClusterFSID 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMisc.ClusterFSID (0 ms) 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMisc.Exec 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMisc.Exec (142 ms) 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMisc.WriteSame 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMisc.WriteSame (22 ms) 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMisc.CmpExt 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMisc.CmpExt (4 ms) 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMisc.Applications 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMisc.Applications (4485 ms) 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatOSD 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatOSD (0 ms) 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMisc.MinCompatClient 2024-09-15T20:33:30.570 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMisc.MinCompatClient (0 ms) 2024-09-15T20:33:30.571 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ RUN ] LibRadosMisc.ShutdownRace 2024-09-15T20:33:30.571 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ OK ] LibRadosMisc.ShutdownRace (24094 ms) 2024-09-15T20:33:30.571 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] 8 tests from LibRadosMisc (28747 ms total) 2024-09-15T20:33:30.571 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: 2024-09-15T20:33:30.571 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [----------] Global test environment tear-down 2024-09-15T20:33:30.571 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [==========] 12 tests from 4 test suites ran. (41890 ms total) 2024-09-15T20:33:30.571 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc: [ PASSED ] 12 tests. 2024-09-15T20:33:39.511 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T20:33:39.512 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T20:33:39.512 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T20:33:39.515 INFO:tasks.ceph.mon.c:Started 2024-09-15T20:33:39.515 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:33:39.515 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:33:39.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient: get_monmap_and_config 2024-09-15T20:33:39.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient: build_initial_monmap 2024-09-15T20:33:39.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient: monmap: 2024-09-15T20:33:39.654 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:39.654 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:39.655 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:39.653710+0000 2024-09-15T20:33:39.655 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:39.653710+0000 2024-09-15T20:33:39.655 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:39.655 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:39.655 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:39.655 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:39.655 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:39.655 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:39.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 1 Processor -- start 2024-09-15T20:33:39.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 1 -- start start 2024-09-15T20:33:39.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient: init 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient(hunting): picked mon.noname-c con 0x7fd614137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient(hunting): picked mon.noname-b con 0x7fd6140a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient(hunting): picked mon.noname-a con 0x7fd61413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6140a6420 con 0x7fd61413e5f0 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6140a6700 con 0x7fd614137ce0 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6140a2110 con 0x7fd6140a9610 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd61aedb640 10 monclient(hunting): authenticate will time out at 1280.538034s 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd619478640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd61413e5f0 0x7fd6141409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:53764/0 (socket says 172.21.15.78:53764) 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd613fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd6140a9610 0x7fd61413ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33008/0 (socket says 172.21.15.78:33008) 2024-09-15T20:33:39.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.650+0000 7fd619478640 1 -- 172.21.15.78:0/3888346396 learned_addr learned my addr 172.21.15.78:0/3888346396 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:39.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134345092 0 0) 0x7fd6140a6420 con 0x7fd61413e5f0 2024-09-15T20:33:39.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient(hunting): _negotiate 2024-09-15T20:33:39.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:39.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:39.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient(hunting): my global_id is 28197 2024-09-15T20:33:39.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd5fc003530 con 0x7fd61413e5f0 2024-09-15T20:33:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1098250892 0 0) 0x7fd5fc003530 con 0x7fd61413e5f0 2024-09-15T20:33:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 >> v1:172.21.15.78:6790/0 conn(0x7fd6140a9610 legacy=0x7fd61413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 >> v1:172.21.15.146:6789/0 conn(0x7fd614137ce0 legacy=0x7fd6141380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient: _finish_hunting 0 2024-09-15T20:33:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 1 monclient: found mon.noname-a 2024-09-15T20:33:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:39.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6140a4bc0 con 0x7fd61413e5f0 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient: _finish_auth 0 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient: _check_auth_tickets 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd60c001a20 con 0x7fd61413e5f0 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd6137fe640 10 monclient: dump: 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:39.695 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd61aedb640 5 monclient: authenticate success, global_id 28197 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.686+0000 7fd61aedb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd60c001bc0 con 0x7fd61413e5f0 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6137fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6137fe640 1 -- 172.21.15.78:0/3888346396 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd60c001f90 con 0x7fd61413e5f0 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 4 set_mon_vals no callback set 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:39.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6137fe640 10 monclient: dump: 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: shutdown 2024-09-15T20:33:39.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- 172.21.15.78:0/3888346396 >> v1:172.21.15.78:6789/0 conn(0x7fd61413e5f0 legacy=0x7fd6141409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- 172.21.15.78:0/3888346396 shutdown_connections 2024-09-15T20:33:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- 172.21.15.78:0/3888346396 >> 172.21.15.78:0/3888346396 conn(0x7fd6140aa6c0 msgr2=0x7fd6140aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- 172.21.15.78:0/3888346396 shutdown_connections 2024-09-15T20:33:39.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- 172.21.15.78:0/3888346396 wait complete. 2024-09-15T20:33:39.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: build_initial_monmap 2024-09-15T20:33:39.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: monmap: 2024-09-15T20:33:39.701 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 librados: starting msgr at 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 librados: starting objecter 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 Processor -- start 2024-09-15T20:33:39.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- start start 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 librados: setting wanted keys 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 librados: calling monclient init 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: init 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient(hunting): picked mon.noname-a con 0x7fd6140a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient(hunting): picked mon.noname-b con 0x7fd614137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient(hunting): picked mon.noname-c con 0x7fd61413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd61413db60 con 0x7fd6140a9610 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6140a0310 con 0x7fd614137ce0 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd6140a05f0 con 0x7fd61413e5f0 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:39.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient(hunting): authenticate will time out at 1280.578593s 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd618c77640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd6140a9610 0x7fd61409cf40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:53768/0 (socket says 172.21.15.78:53768) 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd618c77640 1 -- 172.21.15.78:0/190396540 learned_addr learned my addr 172.21.15.78:0/190396540 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 client.?.objecter ms_handle_connect 0x7fd61413e5f0 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 client.?.objecter ms_handle_connect 0x7fd6140a9610 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1594234027 0 0) 0x7fd61413db60 con 0x7fd6140a9610 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient(hunting): _negotiate 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient(hunting): my global_id is 28200 2024-09-15T20:33:39.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd600004ec0 con 0x7fd6140a9610 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 client.?.objecter ms_handle_connect 0x7fd614137ce0 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 240327173 0 0) 0x7fd600004ec0 con 0x7fd6140a9610 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd61413db60 con 0x7fd6140a9610 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd604001a40 con 0x7fd6140a9610 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient(hunting): dump: 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:39.705 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1656068872 0 0) 0x7fd61413db60 con 0x7fd6140a9610 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 >> v1:172.21.15.78:6790/0 conn(0x7fd61413e5f0 legacy=0x7fd6141d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 >> v1:172.21.15.146:6789/0 conn(0x7fd614137ce0 legacy=0x7fd61409d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient: _finish_hunting 0 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 monclient: found mon.a 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd61413e370 con 0x7fd6140a9610 2024-09-15T20:33:39.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient: _finish_auth 0 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient: _check_auth_tickets 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 5 monclient: authenticate success, global_id 28200 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: _renew_subs 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd6140a14b0 con 0x7fd6140a9610 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 client.28200.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: _renew_subs 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd61409dd60 con 0x7fd6140a9610 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd61aedb640 1 librados: init done 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd604001d00 con 0x7fd6140a9610 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:39.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd604003a70 con 0x7fd6140a9610 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd6117fa640 10 monclient: dump: 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:39.708 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.690+0000 7fd612ffd640 4 set_mon_vals no callback set 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd61aedb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd61aedb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd61aedb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5f4005090 con 0x7fd6140a9610 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd604003c10 con 0x7fd6140a9610 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(72..72 src has 1..72) ==== 7591+0+0 (unknown 1390996731 0 0) 0x7fd60404c130 con 0x7fd6140a9610 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd6117fa640 10 client.28200.objecter ms_dispatch 0x7fd614000ba0 osd_map(72..72 src has 1..72) 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd6117fa640 3 client.28200.objecter handle_osd_map got epochs [72,72] > 0 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd6117fa640 3 client.28200.objecter handle_osd_map decoding full epoch 72 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd6117fa640 20 client.28200.objecter dump_active .. 0 homeless 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd613fff640 10 monclient: get_auth_request con 0x7fd60003e030 auth_method 0 2024-09-15T20:33:39.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.694+0000 7fd6117fa640 10 client.28200.objecter ms_handle_connect 0x7fd60003e030 2024-09-15T20:33:39.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.710+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd604016ab0 con 0x7fd6140a9610 2024-09-15T20:33:39.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.710+0000 7fd6117fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:39.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.710+0000 7fd6117fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.950+0000 7fd61aedb640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:33:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.950+0000 7fd61aedb640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.950+0000 7fd61aedb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.950+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd5f4005380 con 0x7fd6140a9610 2024-09-15T20:33:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd6117fa640 1 -- 172.21.15.78:0/190396540 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 311937782) 0x7fd60401ae90 con 0x7fd6140a9610 2024-09-15T20:33:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd6117fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd6117fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:39.958 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:33:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 10 librados: watch_flush enter 2024-09-15T20:33:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 10 librados: watch_flush exit 2024-09-15T20:33:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 20 client.28200.objecter shutdown clearing up homeless session... 2024-09-15T20:33:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 10 client.28200.objecter successfully canceled tick 2024-09-15T20:33:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 >> v1:172.21.15.146:6812/435598642 conn(0x7fd60003e030 legacy=0x7fd6000404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 10 monclient: shutdown 2024-09-15T20:33:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 >> v1:172.21.15.78:6789/0 conn(0x7fd6140a9610 legacy=0x7fd61409cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 shutdown_connections 2024-09-15T20:33:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 >> 172.21.15.78:0/190396540 conn(0x7fd6140aa6c0 msgr2=0x7fd614130cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 shutdown_connections 2024-09-15T20:33:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 1 -- 172.21.15.78:0/190396540 wait complete. 2024-09-15T20:33:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:39.954+0000 7fd61aedb640 1 librados: shutdown 2024-09-15T20:33:39.976 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:33:39.976 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:33:39.976 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: get_monmap_and_config 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: build_initial_monmap 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: monmap: 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:40.089092+0000 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:40.089092+0000 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.090 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:40.091 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:40.091 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:40.091 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 Processor -- start 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- start start 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: init 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient(hunting): picked mon.noname-b con 0x7f71d8137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient(hunting): picked mon.noname-c con 0x7f71d80a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient(hunting): picked mon.noname-a con 0x7f71d813e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d80a6420 con 0x7f71d813e5f0 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d80a6700 con 0x7f71d80a9610 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d80a2110 con 0x7f71d8137ce0 2024-09-15T20:33:40.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient(hunting): authenticate will time out at 1280.973641s 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71debfc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f71d813e5f0 0x7f71d81409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:53774/0 (socket says 172.21.15.78:53774) 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71debfc640 1 -- 172.21.15.78:0/219823722 learned_addr learned my addr 172.21.15.78:0/219823722 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71de3fb640 1 --1- 172.21.15.78:0/219823722 >> v1:172.21.15.78:6790/0 conn(0x7f71d8137ce0 0x7f71d81380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33022/0 (socket says 172.21.15.78:33022) 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308729728 0 0) 0x7f71d80a6420 con 0x7f71d813e5f0 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient(hunting): my global_id is 28203 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71c0003530 con 0x7f71d813e5f0 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2129378030 0 0) 0x7f71d80a2110 con 0x7f71d8137ce0 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient(hunting): my global_id is 74099 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d80a6420 con 0x7f71d8137ce0 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2743489837 0 0) 0x7f71d80a6420 con 0x7f71d8137ce0 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 >> v1:172.21.15.146:6789/0 conn(0x7f71d80a9610 legacy=0x7f71d813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 >> v1:172.21.15.78:6789/0 conn(0x7f71d813e5f0 legacy=0x7f71d81409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: _finish_hunting 0 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 monclient: found mon.noname-b 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d80a4bc0 con 0x7f71d8137ce0 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: _finish_auth 0 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: _check_auth_tickets 2024-09-15T20:33:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 5 monclient: authenticate success, global_id 74099 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71c8001680 con 0x7f71d8137ce0 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: dump: 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71c8001820 con 0x7f71d8137ce0 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 1 -- 172.21.15.78:0/219823722 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71c8001c30 con 0x7f71d8137ce0 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71dd3f9640 10 monclient: dump: 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.103 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 4 set_mon_vals no callback set 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: shutdown 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- 172.21.15.78:0/219823722 >> v1:172.21.15.78:6790/0 conn(0x7f71d8137ce0 legacy=0x7f71d81380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- 172.21.15.78:0/219823722 shutdown_connections 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- 172.21.15.78:0/219823722 >> 172.21.15.78:0/219823722 conn(0x7f71d80aa6c0 msgr2=0x7f71d80aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- 172.21.15.78:0/219823722 shutdown_connections 2024-09-15T20:33:40.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 1 -- 172.21.15.78:0/219823722 wait complete. 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: build_initial_monmap 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.086+0000 7f71e065f640 10 monclient: monmap: 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 librados: starting msgr at 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 librados: starting objecter 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 Processor -- start 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 -- start start 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 librados: setting wanted keys 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 librados: calling monclient init 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient: init 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient(hunting): picked mon.noname-b con 0x7f71d80a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient(hunting): picked mon.noname-a con 0x7f71d8137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient(hunting): picked mon.noname-c con 0x7f71d813e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d813c0e0 con 0x7f71d8137ce0 2024-09-15T20:33:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d80a9200 con 0x7f71d80a9610 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71d81cd500 con 0x7f71d813e5f0 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient(hunting): authenticate will time out at 1280.976855s 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ddbfa640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f71d8137ce0 0x7f71d81ccdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:53780/0 (socket says 172.21.15.78:53780) 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ddbfa640 1 -- 172.21.15.78:0/910027333 learned_addr learned my addr 172.21.15.78:0/910027333 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.?.objecter ms_handle_connect 0x7f71d813e5f0 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.?.objecter ms_handle_connect 0x7f71d8137ce0 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3230336702 0 0) 0x7f71d81cd500 con 0x7f71d813e5f0 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): my global_id is 74102 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71c4004ec0 con 0x7f71d813e5f0 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 930889488 0 0) 0x7f71d813c0e0 con 0x7f71d8137ce0 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): my global_id is 28206 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d81cd500 con 0x7f71d8137ce0 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.?.objecter ms_handle_connect 0x7f71d80a9610 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2811551775 0 0) 0x7f71d81cd500 con 0x7f71d8137ce0 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71d813c0e0 con 0x7f71d8137ce0 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71d40018e0 con 0x7f71d8137ce0 2024-09-15T20:33:40.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): dump: 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1363131513 0 0) 0x7f71c4004ec0 con 0x7f71d813e5f0 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71d81cd500 con 0x7f71d813e5f0 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71d00014c0 con 0x7f71d813e5f0 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): dump: 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1444332148 0 0) 0x7f71d80a9200 con 0x7f71d80a9610 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): my global_id is 22702 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71c4004ec0 con 0x7f71d80a9610 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3407842073 0 0) 0x7f71d813c0e0 con 0x7f71d8137ce0 2024-09-15T20:33:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 >> v1:172.21.15.78:6790/0 conn(0x7f71d813e5f0 legacy=0x7f71d81d4c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 >> v1:172.21.15.146:6789/0 conn(0x7f71d80a9610 legacy=0x7f71d81cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: _finish_hunting 0 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 monclient: found mon.a 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71d8139fe0 con 0x7f71d8137ce0 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: _finish_auth 0 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: _check_auth_tickets 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 5 monclient: authenticate success, global_id 28206 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient: _renew_subs 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f71d80a14b0 con 0x7f71d8137ce0 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 client.28206.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient: _renew_subs 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f71d813a8e0 con 0x7f71d8137ce0 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 librados: init done 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f71d4003620 con 0x7f71d8137ce0 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71d4003a30 con 0x7f71d8137ce0 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71dcbf8640 4 set_mon_vals no callback set 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: dump: 2024-09-15T20:33:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71b80065d0 con 0x7f71b8002870 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.28206.objecter ms_handle_connect 0x7f71b8002870 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.28206.objecter resend_mon_ops 2024-09-15T20:33:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f71d4003bd0 con 0x7f71d8137ce0 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 230090820 0 0) 0x7f71b80065d0 con 0x7f71b8002870 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient(hunting): my global_id is 28209 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d813c0e0 con 0x7f71b8002870 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(72..72 src has 1..72) ==== 7591+0+0 (unknown 1390996731 0 0) 0x7f71d404bfb0 con 0x7f71d8137ce0 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.28206.objecter ms_dispatch 0x7f71d8000ba0 osd_map(72..72 src has 1..72) 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 3 client.28206.objecter handle_osd_map got epochs [72,72] > 0 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 3 client.28206.objecter handle_osd_map decoding full epoch 72 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 20 client.28206.objecter dump_active .. 0 homeless 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3311946798 0 0) 0x7f71d813c0e0 con 0x7f71b8002870 2024-09-15T20:33:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71b80065d0 con 0x7f71b8002870 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71c8001710 con 0x7f71b8002870 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: dump: 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71debfc640 10 monclient: get_auth_request con 0x7f71c403e3e0 auth_method 0 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2974673785 0 0) 0x7f71b80065d0 con 0x7f71b8002870 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71b800a170 con 0x7f71b8002870 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 client.28206.objecter ms_handle_connect 0x7f71c403e3e0 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f71c8001570 con 0x7f71b8002870 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 >> v1:172.21.15.78:6789/0 conn(0x7f71b8002870 legacy=0x7f71b8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.090+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 >> v1:172.21.15.78:6789/0 conn(0x7f71b8002870 legacy=0x7f71b8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f71b8009e80 con 0x7f71b8009140 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 client.28206.objecter ms_handle_connect 0x7f71b8009140 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 client.28206.objecter resend_mon_ops 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2453974505 0 0) 0x7f71b8009e80 con 0x7f71b8009140 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient(hunting): my global_id is 28212 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71b80065d0 con 0x7f71b8009140 2024-09-15T20:33:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2287740769 0 0) 0x7f71b80065d0 con 0x7f71b8009140 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f71b8009e80 con 0x7f71b8009140 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f71c8004b80 con 0x7f71b8009140 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient: dump: 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3022687969 0 0) 0x7f71b8009e80 con 0x7f71b8009140 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71b800c420 con 0x7f71b8009140 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4140 (unknown 0 0 3018299933) 0x7f71c8006010 con 0x7f71b8009140 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 >> v1:172.21.15.78:6789/0 conn(0x7f71b8009140 legacy=0x7f71b80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71ceffd640 1 -- 172.21.15.78:0/910027333 >> v1:172.21.15.78:6789/0 conn(0x7f71b8009140 legacy=0x7f71b80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:40.121 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 10, 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 132649, 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:33:40.122 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.123 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.124 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:40.125 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.126 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:33:40.127 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:40.128 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.129 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:33:40.135 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:33:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 10 librados: watch_flush enter 2024-09-15T20:33:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 10 librados: watch_flush exit 2024-09-15T20:33:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 20 client.28206.objecter shutdown clearing up homeless session... 2024-09-15T20:33:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 10 client.28206.objecter successfully canceled tick 2024-09-15T20:33:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 >> v1:172.21.15.146:6812/435598642 conn(0x7f71c403e3e0 legacy=0x7f71c40408a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 10 monclient: shutdown 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 >> v1:172.21.15.78:6789/0 conn(0x7f71d8137ce0 legacy=0x7f71d81ccdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71debfc640 1 -- 172.21.15.78:0/910027333 reap_dead start 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71debfc640 1 -- 172.21.15.78:0/910027333 reap_dead start 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 shutdown_connections 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 >> 172.21.15.78:0/910027333 conn(0x7f71d80aa6c0 msgr2=0x7f71d8073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 shutdown_connections 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 1 -- 172.21.15.78:0/910027333 wait complete. 2024-09-15T20:33:40.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.098+0000 7f71e065f640 1 librados: shutdown 2024-09-15T20:33:40.138 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:33:40.138 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:33:40.139 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:33:40.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 10 monclient: get_monmap_and_config 2024-09-15T20:33:40.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 10 monclient: build_initial_monmap 2024-09-15T20:33:40.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 10 monclient: monmap: 2024-09-15T20:33:40.256 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:40.256 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.256 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:40.255619+0000 2024-09-15T20:33:40.257 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:40.255619+0000 2024-09-15T20:33:40.257 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:40.257 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.257 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:40.257 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:40.257 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:40.257 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 1 Processor -- start 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 1 -- start start 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 10 monclient: init 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 10 monclient(hunting): picked mon.noname-a con 0x7f749c135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.250+0000 7f74a16fa640 10 monclient(hunting): picked mon.noname-b con 0x7f749c0a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient(hunting): picked mon.noname-c con 0x7f749c13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749c0745e0 con 0x7f749c135b60 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749c06ecf0 con 0x7f749c13c220 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749c06efd0 con 0x7f749c0a24e0 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:40.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient(hunting): authenticate will time out at 1281.140059s 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f749a7fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f749c0a24e0 0x7f749c13ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:42992/0 (socket says 172.21.15.78:42992) 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f749a7fc640 1 -- 172.21.15.78:0/3315471096 learned_addr learned my addr 172.21.15.78:0/3315471096 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1711125556 0 0) 0x7f749c06efd0 con 0x7f749c0a24e0 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient(hunting): my global_id is 74105 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7484003530 con 0x7f749c0a24e0 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 817555568 0 0) 0x7f749c0745e0 con 0x7f749c135b60 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient(hunting): my global_id is 28215 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f749c06efd0 con 0x7f749c135b60 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 335920955 0 0) 0x7f7484003530 con 0x7f749c0a24e0 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 >> v1:172.21.15.146:6789/0 conn(0x7f749c13c220 legacy=0x7f749c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 >> v1:172.21.15.78:6789/0 conn(0x7f749c135b60 legacy=0x7f749c135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 monclient: found mon.noname-b 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749c136650 con 0x7f749c0a24e0 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: _finish_auth 0 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 5 monclient: authenticate success, global_id 74105 2024-09-15T20:33:40.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f74900016a0 con 0x7f749c0a24e0 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: dump: 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.269 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7490001840 con 0x7f749c0a24e0 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 1 -- 172.21.15.78:0/3315471096 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7490001c10 con 0x7f749c0a24e0 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f7499ffb640 10 monclient: dump: 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.270 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 4 set_mon_vals no callback set 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient: shutdown 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- 172.21.15.78:0/3315471096 >> v1:172.21.15.78:6790/0 conn(0x7f749c0a24e0 legacy=0x7f749c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- 172.21.15.78:0/3315471096 shutdown_connections 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- 172.21.15.78:0/3315471096 >> 172.21.15.78:0/3315471096 conn(0x7f749c0a63c0 msgr2=0x7f749c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- 172.21.15.78:0/3315471096 shutdown_connections 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- 172.21.15.78:0/3315471096 wait complete. 2024-09-15T20:33:40.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient: build_initial_monmap 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient: monmap: 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 librados: starting msgr at 2024-09-15T20:33:40.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 librados: starting objecter 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 Processor -- start 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- start start 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 librados: setting wanted keys 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 librados: calling monclient init 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient: init 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient(hunting): picked mon.noname-c con 0x7f749c0a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient(hunting): picked mon.noname-b con 0x7f749c135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient(hunting): picked mon.noname-a con 0x7f749c13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749c073700 con 0x7f749c13c220 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749c072920 con 0x7f749c135b60 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f749c09d180 con 0x7f749c0a24e0 2024-09-15T20:33:40.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74a16fa640 10 monclient(hunting): authenticate will time out at 1281.143231s 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f749affd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f749c0a24e0 0x7f749c071b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:42996/0 (socket says 172.21.15.78:42996) 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f749affd640 1 -- 172.21.15.78:0/306930032 learned_addr learned my addr 172.21.15.78:0/306930032 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f749b7fe640 1 --1- 172.21.15.78:0/306930032 >> v1:172.21.15.78:6789/0 conn(0x7f749c13c220 0x7f749c1d4eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36858/0 (socket says 172.21.15.78:36858) 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 client.?.objecter ms_handle_connect 0x7f749c13c220 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 client.?.objecter ms_handle_connect 0x7f749c0a24e0 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3834449116 0 0) 0x7f749c073700 con 0x7f749c13c220 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): my global_id is 28218 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7488004ec0 con 0x7f749c13c220 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2690704017 0 0) 0x7f749c09d180 con 0x7f749c0a24e0 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): my global_id is 74108 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f749c073700 con 0x7f749c0a24e0 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 client.?.objecter ms_handle_connect 0x7f749c135b60 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 885755309 0 0) 0x7f7488004ec0 con 0x7f749c13c220 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f749c09d180 con 0x7f749c13c220 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7494001c80 con 0x7f749c13c220 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:40.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): dump: 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.254+0000 7f74837fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:40.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3246603411 0 0) 0x7f749c072920 con 0x7f749c135b60 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient(hunting): my global_id is 22708 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7488004ec0 con 0x7f749c135b60 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3770602284 0 0) 0x7f749c09d180 con 0x7f749c13c220 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 >> v1:172.21.15.78:6790/0 conn(0x7f749c0a24e0 legacy=0x7f749c071b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 >> v1:172.21.15.146:6789/0 conn(0x7f749c135b60 legacy=0x7f749c072210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: _finish_hunting 0 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 monclient: found mon.a 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f749c13be50 con 0x7f749c13c220 2024-09-15T20:33:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: _finish_auth 0 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: _check_auth_tickets 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 5 monclient: authenticate success, global_id 28218 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 10 monclient: _renew_subs 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f749c09e390 con 0x7f749c13c220 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 10 client.28218.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 10 monclient: _renew_subs 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f749c13b7c0 con 0x7f749c13c220 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 1 librados: init done 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f74940039a0 con 0x7f749c13c220 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:40.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7494003d30 con 0x7f749c13c220 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74997fa640 4 set_mon_vals no callback set 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: dump: 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.279 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f747c0065d0 con 0x7f747c002870 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 client.28218.objecter ms_handle_connect 0x7f747c002870 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 client.28218.objecter resend_mon_ops 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4081254264 0 0) 0x7f747c0065d0 con 0x7f747c002870 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient(hunting): my global_id is 74111 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f749c09d180 con 0x7f747c002870 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f7494003ed0 con 0x7f749c13c220 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(72..72 src has 1..72) ==== 7591+0+0 (unknown 1390996731 0 0) 0x7f749404c370 con 0x7f749c13c220 2024-09-15T20:33:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 client.28218.objecter ms_dispatch 0x7f749c000ba0 osd_map(72..72 src has 1..72) 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 3 client.28218.objecter handle_osd_map got epochs [72,72] > 0 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 3 client.28218.objecter handle_osd_map decoding full epoch 72 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 20 client.28218.objecter dump_active .. 0 homeless 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1874294000 0 0) 0x7f749c09d180 con 0x7f747c002870 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f747c0065d0 con 0x7f747c002870 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f748c0037a0 con 0x7f747c002870 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: dump: 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f749a7fc640 10 monclient: get_auth_request con 0x7f748803df80 auth_method 0 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2950502642 0 0) 0x7f747c0065d0 con 0x7f747c002870 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f747c00a170 con 0x7f747c002870 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 client.28218.objecter ms_handle_connect 0x7f748803df80 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f748c003600 con 0x7f747c002870 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 >> v1:172.21.15.78:6790/0 conn(0x7f747c002870 legacy=0x7f747c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.258+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 >> v1:172.21.15.78:6790/0 conn(0x7f747c002870 legacy=0x7f747c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f747c009e80 con 0x7f747c009140 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 client.28218.objecter ms_handle_connect 0x7f747c009140 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 client.28218.objecter resend_mon_ops 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2934607350 0 0) 0x7f747c009e80 con 0x7f747c009140 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient(hunting): my global_id is 74114 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f747c0065d0 con 0x7f747c009140 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1620733117 0 0) 0x7f747c0065d0 con 0x7f747c009140 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f747c009e80 con 0x7f747c009140 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f748c0041e0 con 0x7f747c009140 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient: dump: 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.285 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 919383549 0 0) 0x7f747c009e80 con 0x7f747c009140 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f747c00c400 con 0x7f747c009140 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 84291892) 0x7f748c004260 con 0x7f747c009140 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 >> v1:172.21.15.78:6790/0 conn(0x7f747c009140 legacy=0x7f747c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74837fe640 1 -- 172.21.15.78:0/306930032 >> v1:172.21.15.78:6790/0 conn(0x7f747c009140 legacy=0x7f747c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 10, 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 663, 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:33:40.287 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.288 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.289 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:40.290 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.291 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.292 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.293 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:33:40.294 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.295 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.296 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:33:40.297 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:33:40.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 10 librados: watch_flush enter 2024-09-15T20:33:40.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 10 librados: watch_flush exit 2024-09-15T20:33:40.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 20 client.28218.objecter shutdown clearing up homeless session... 2024-09-15T20:33:40.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 10 client.28218.objecter successfully canceled tick 2024-09-15T20:33:40.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 >> v1:172.21.15.146:6812/435598642 conn(0x7f748803df80 legacy=0x7f7488040440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 10 monclient: shutdown 2024-09-15T20:33:40.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:40.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 >> v1:172.21.15.78:6789/0 conn(0x7f749c13c220 legacy=0x7f749c1d4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f749b7fe640 1 -- 172.21.15.78:0/306930032 reap_dead start 2024-09-15T20:33:40.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f749b7fe640 1 -- 172.21.15.78:0/306930032 reap_dead start 2024-09-15T20:33:40.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 shutdown_connections 2024-09-15T20:33:40.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 >> 172.21.15.78:0/306930032 conn(0x7f749c0a63c0 msgr2=0x7f749c0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:40.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 shutdown_connections 2024-09-15T20:33:40.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 1 -- 172.21.15.78:0/306930032 wait complete. 2024-09-15T20:33:40.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.266+0000 7f74a16fa640 1 librados: shutdown 2024-09-15T20:33:40.300 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:33:40.300 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:33:40.301 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient: get_monmap_and_config 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient: build_initial_monmap 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient: monmap: 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:40.417345+0000 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:40.417345+0000 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.418 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:40.419 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:40.419 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:40.419 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 Processor -- start 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- start start 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient: init 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient(hunting): picked mon.noname-c con 0x7fa14009d950 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient(hunting): picked mon.noname-b con 0x7fa140074630 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient(hunting): picked mon.noname-a con 0x7fa140074a30 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa14009f900 con 0x7fa140074a30 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa14009fbe0 con 0x7fa14009d950 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa14009fec0 con 0x7fa140074630 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:40.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient(hunting): authenticate will time out at 1281.301952s 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa145062640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa140074630 0x7fa14013a230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43012/0 (socket says 172.21.15.78:43012) 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa145062640 1 -- 172.21.15.78:0/3268503825 learned_addr learned my addr 172.21.15.78:0/3268503825 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa146064640 1 --1- 172.21.15.78:0/3268503825 >> v1:172.21.15.78:6789/0 conn(0x7fa140074a30 0x7fa14013d960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36872/0 (socket says 172.21.15.78:36872) 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 533000375 0 0) 0x7fa14009fec0 con 0x7fa140074630 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient(hunting): my global_id is 74117 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa128003530 con 0x7fa140074630 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 324393731 0 0) 0x7fa14009f900 con 0x7fa140074a30 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient(hunting): my global_id is 28221 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa14009fec0 con 0x7fa140074a30 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3914404100 0 0) 0x7fa14009fec0 con 0x7fa140074a30 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 >> v1:172.21.15.78:6790/0 conn(0x7fa140074630 legacy=0x7fa14013a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 >> v1:172.21.15.146:6789/0 conn(0x7fa14009d950 legacy=0x7fa1400a0d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: _finish_hunting 0 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 monclient: found mon.noname-a 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa140074ea0 con 0x7fa140074a30 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: _finish_auth 0 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: _check_auth_tickets 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 5 monclient: authenticate success, global_id 28221 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa138001680 con 0x7fa140074a30 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: dump: 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa1380018a0 con 0x7fa140074a30 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 1 -- 172.21.15.78:0/3268503825 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa138001cb0 con 0x7fa140074a30 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 4 set_mon_vals no callback set 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa144861640 10 monclient: dump: 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 10 monclient: shutdown 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- 172.21.15.78:0/3268503825 >> v1:172.21.15.78:6789/0 conn(0x7fa140074a30 legacy=0x7fa14013d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- 172.21.15.78:0/3268503825 shutdown_connections 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- 172.21.15.78:0/3268503825 >> 172.21.15.78:0/3268503825 conn(0x7fa1400a63c0 msgr2=0x7fa1400a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- 172.21.15.78:0/3268503825 shutdown_connections 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.414+0000 7fa147ac7640 1 -- 172.21.15.78:0/3268503825 wait complete. 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: build_initial_monmap 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: monmap: 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 librados: starting msgr at 2024-09-15T20:33:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 librados: starting objecter 2024-09-15T20:33:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 Processor -- start 2024-09-15T20:33:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 -- start start 2024-09-15T20:33:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 librados: setting wanted keys 2024-09-15T20:33:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 librados: calling monclient init 2024-09-15T20:33:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: init 2024-09-15T20:33:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:33:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient(hunting): picked mon.noname-a con 0x7fa140074630 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient(hunting): picked mon.noname-b con 0x7fa140074a30 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient(hunting): picked mon.noname-c con 0x7fa14009d950 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa1401302b0 con 0x7fa140074630 2024-09-15T20:33:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa140071670 con 0x7fa140074a30 2024-09-15T20:33:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa140071950 con 0x7fa14009d950 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient(hunting): authenticate will time out at 1281.304997s 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa145863640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa140074630 0x7fa140073830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36876/0 (socket says 172.21.15.78:36876) 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa145863640 1 -- 172.21.15.78:0/967964655 learned_addr learned my addr 172.21.15.78:0/967964655 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.?.objecter ms_handle_connect 0x7fa140074630 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.?.objecter ms_handle_connect 0x7fa14009d950 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3845964653 0 0) 0x7fa1401302b0 con 0x7fa140074630 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): my global_id is 28224 2024-09-15T20:33:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa12c004ec0 con 0x7fa140074630 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1450018766 0 0) 0x7fa140071950 con 0x7fa14009d950 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): my global_id is 74120 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1401302b0 con 0x7fa14009d950 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.?.objecter ms_handle_connect 0x7fa140074a30 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4233360308 0 0) 0x7fa1401302b0 con 0x7fa14009d950 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa140071950 con 0x7fa14009d950 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa138001bf0 con 0x7fa14009d950 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): dump: 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3800327934 0 0) 0x7fa12c004ec0 con 0x7fa140074630 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1401302b0 con 0x7fa140074630 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa130001d40 con 0x7fa140074630 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): dump: 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2420495382 0 0) 0x7fa140071950 con 0x7fa14009d950 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 >> v1:172.21.15.146:6789/0 conn(0x7fa140074a30 legacy=0x7fa1401d1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 >> v1:172.21.15.78:6789/0 conn(0x7fa140074630 legacy=0x7fa140073830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 monclient: found mon.c 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa140070710 con 0x7fa14009d950 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient: _finish_auth 0 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 5 monclient: authenticate success, global_id 74120 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: _renew_subs 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa1400702f0 con 0x7fa14009d950 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 client.74120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: _renew_subs 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa14012e240 con 0x7fa14009d950 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 librados: init done 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa138001800 con 0x7fa14009d950 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa138004b40 con 0x7fa14009d950 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13ffff640 4 set_mon_vals no callback set 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient: dump: 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fa138010bb0 con 0x7fa14009d950 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa1200065d0 con 0x7fa120002870 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(72..72 src has 1..72) ==== 7591+0+0 (unknown 1390996731 0 0) 0x7fa13804bed0 con 0x7fa14009d950 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.74120.objecter ms_dispatch 0x7fa140000ba0 osd_map(72..72 src has 1..72) 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 3 client.74120.objecter handle_osd_map got epochs [72,72] > 0 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 3 client.74120.objecter handle_osd_map decoding full epoch 72 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 20 client.74120.objecter dump_active .. 0 homeless 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.74120.objecter ms_handle_connect 0x7fa120002870 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.74120.objecter resend_mon_ops 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa145062640 10 monclient: get_auth_request con 0x7fa12c03e040 auth_method 0 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030152148 0 0) 0x7fa1200065d0 con 0x7fa120002870 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 monclient(hunting): my global_id is 22717 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa140071950 con 0x7fa120002870 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.418+0000 7fa13e7fc640 10 client.74120.objecter ms_handle_connect 0x7fa12c03e040 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2413893685 0 0) 0x7fa140071950 con 0x7fa120002870 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa1200065d0 con 0x7fa120002870 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa130003be0 con 0x7fa120002870 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 10 monclient: dump: 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3459033480 0 0) 0x7fa1200065d0 con 0x7fa120002870 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa12000a170 con 0x7fa120002870 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa130003d50 con 0x7fa120002870 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 >> v1:172.21.15.146:6789/0 conn(0x7fa120002870 legacy=0x7fa120004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.422+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 >> v1:172.21.15.146:6789/0 conn(0x7fa120002870 legacy=0x7fa120004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.426+0000 7fa147ac7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.426+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa120009e60 con 0x7fa120009120 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.426+0000 7fa13e7fc640 10 client.74120.objecter ms_handle_connect 0x7fa120009120 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.426+0000 7fa13e7fc640 10 client.74120.objecter resend_mon_ops 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 644094095 0 0) 0x7fa120009e60 con 0x7fa120009120 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient(hunting): my global_id is 22720 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa1200065d0 con 0x7fa120009120 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2493411770 0 0) 0x7fa1200065d0 con 0x7fa120009120 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa120009e60 con 0x7fa120009120 2024-09-15T20:33:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa130004460 con 0x7fa120009120 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient: dump: 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 754650780 0 0) 0x7fa120009e60 con 0x7fa120009120 2024-09-15T20:33:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa12000c3b0 con 0x7fa120009120 2024-09-15T20:33:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4306 (unknown 0 0 1982220504) 0x7fa130004280 con 0x7fa120009120 2024-09-15T20:33:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:33:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:40.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 >> v1:172.21.15.146:6789/0 conn(0x7fa120009120 legacy=0x7fa1200095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa13e7fc640 1 -- 172.21.15.78:0/967964655 >> v1:172.21.15.146:6789/0 conn(0x7fa120009120 legacy=0x7fa1200095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 10, 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 132938, 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:33:40.448 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.449 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.450 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:33:40.451 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:40.452 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:33:40.453 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:33:40.454 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.455 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:40.456 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.457 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.458 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 10 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:40.459 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:40.460 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:40.460 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:40.460 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:40.460 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:33:40.460 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:33:40.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 10 librados: watch_flush enter 2024-09-15T20:33:40.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 10 librados: watch_flush exit 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 20 client.74120.objecter shutdown clearing up homeless session... 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 10 client.74120.objecter successfully canceled tick 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 >> v1:172.21.15.146:6812/435598642 conn(0x7fa12c03e040 legacy=0x7fa12c040500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 10 monclient: shutdown 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 >> v1:172.21.15.78:6790/0 conn(0x7fa14009d950 legacy=0x7fa1401d4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa146064640 1 -- 172.21.15.78:0/967964655 reap_dead start 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa146064640 1 -- 172.21.15.78:0/967964655 reap_dead start 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 shutdown_connections 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 >> 172.21.15.78:0/967964655 conn(0x7fa1400a63c0 msgr2=0x7fa1400a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 shutdown_connections 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 1 -- 172.21.15.78:0/967964655 wait complete. 2024-09-15T20:33:40.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.430+0000 7fa147ac7640 1 librados: shutdown 2024-09-15T20:33:40.463 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:33:40.463 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:33:40.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:33:40.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient: get_monmap_and_config 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient: build_initial_monmap 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient: monmap: 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:40.587149+0000 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:40.587149+0000 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:40.588 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 1 Processor -- start 2024-09-15T20:33:40.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 1 -- start start 2024-09-15T20:33:40.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient: init 2024-09-15T20:33:40.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:40.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:33:40.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient(hunting): picked mon.noname-b con 0x7fbe90137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:40.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient(hunting): picked mon.noname-a con 0x7fbe90074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:40.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient(hunting): picked mon.noname-c con 0x7fbe9013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe90138850 con 0x7fbe90074500 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe900a5da0 con 0x7fbe9013e370 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe90073560 con 0x7fbe90137d60 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe95e19640 10 monclient(hunting): authenticate will time out at 1281.471400s 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe87fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbe90074500 0x7fbe9013cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36878/0 (socket says 172.21.15.78:36878) 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe87fff640 1 -- 172.21.15.78:0/449678405 learned_addr learned my addr 172.21.15.78:0/449678405 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2895568951 0 0) 0x7fbe90138850 con 0x7fbe90074500 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 10 monclient(hunting): my global_id is 28227 2024-09-15T20:33:40.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe7c003530 con 0x7fbe90074500 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1209255724 0 0) 0x7fbe90073560 con 0x7fbe90137d60 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 10 monclient(hunting): my global_id is 74123 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.582+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe90138850 con 0x7fbe90137d60 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1982579615 0 0) 0x7fbe7c003530 con 0x7fbe90074500 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 >> v1:172.21.15.78:6790/0 conn(0x7fbe90137d60 legacy=0x7fbe90138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 >> v1:172.21.15.146:6789/0 conn(0x7fbe9013e370 legacy=0x7fbe90140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: _finish_hunting 0 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 1 monclient: found mon.noname-a 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:40.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe90073840 con 0x7fbe90074500 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: _finish_auth 0 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: _check_auth_tickets 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 5 monclient: authenticate success, global_id 28227 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbe780016a0 con 0x7fbe90074500 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: dump: 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.594 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbe78001840 con 0x7fbe90074500 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 1 -- 172.21.15.78:0/449678405 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbe78001c50 con 0x7fbe90074500 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 4 set_mon_vals no callback set 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:40.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8effd640 10 monclient: dump: 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: shutdown 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- 172.21.15.78:0/449678405 >> v1:172.21.15.78:6789/0 conn(0x7fbe90074500 legacy=0x7fbe9013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- 172.21.15.78:0/449678405 shutdown_connections 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- 172.21.15.78:0/449678405 >> 172.21.15.78:0/449678405 conn(0x7fbe900aa650 msgr2=0x7fbe900aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- 172.21.15.78:0/449678405 shutdown_connections 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- 172.21.15.78:0/449678405 wait complete. 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: build_initial_monmap 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: monmap: 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:40.597 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 librados: starting msgr at 2024-09-15T20:33:40.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 librados: starting objecter 2024-09-15T20:33:40.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 Processor -- start 2024-09-15T20:33:40.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- start start 2024-09-15T20:33:40.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 librados: setting wanted keys 2024-09-15T20:33:40.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 librados: calling monclient init 2024-09-15T20:33:40.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: init 2024-09-15T20:33:40.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient(hunting): picked mon.noname-c con 0x7fbe90074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient(hunting): picked mon.noname-b con 0x7fbe90137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient(hunting): picked mon.noname-a con 0x7fbe9013e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe90135970 con 0x7fbe9013e370 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe90132d50 con 0x7fbe90137d60 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe90134450 con 0x7fbe90074500 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient(hunting): authenticate will time out at 1281.474340s 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8f7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fbe90074500 0x7fbe90131f30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43038/0 (socket says 172.21.15.78:43038) 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbe9013e370 0x7fbe901d45c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36892/0 (socket says 172.21.15.78:36892) 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8f7fe640 1 -- 172.21.15.78:0/3351237804 learned_addr learned my addr 172.21.15.78:0/3351237804 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:40.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 client.?.objecter ms_handle_connect 0x7fbe9013e370 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 client.?.objecter ms_handle_connect 0x7fbe90074500 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1988282815 0 0) 0x7fbe90135970 con 0x7fbe9013e370 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): my global_id is 28230 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe70004ef0 con 0x7fbe9013e370 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405550785 0 0) 0x7fbe90134450 con 0x7fbe90074500 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): my global_id is 74126 2024-09-15T20:33:40.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe90135970 con 0x7fbe90074500 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 client.?.objecter ms_handle_connect 0x7fbe90137d60 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 39004783 0 0) 0x7fbe70004ef0 con 0x7fbe9013e370 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe90134450 con 0x7fbe9013e370 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbe88001c60 con 0x7fbe9013e370 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): dump: 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.604 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2298937857 0 0) 0x7fbe90135970 con 0x7fbe90074500 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbe70004ef0 con 0x7fbe90074500 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbe800013c0 con 0x7fbe90074500 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): dump: 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.605 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2942534854 0 0) 0x7fbe90132d50 con 0x7fbe90137d60 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _negotiate 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient(hunting): my global_id is 22726 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbe90135970 con 0x7fbe90137d60 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2578506783 0 0) 0x7fbe90134450 con 0x7fbe9013e370 2024-09-15T20:33:40.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 >> v1:172.21.15.78:6790/0 conn(0x7fbe90074500 legacy=0x7fbe90131f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 >> v1:172.21.15.146:6789/0 conn(0x7fbe90137d60 legacy=0x7fbe90132640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient: _finish_hunting 0 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 monclient: found mon.a 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe90141960 con 0x7fbe9013e370 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient: _finish_auth 0 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient: _check_auth_tickets 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 5 monclient: authenticate success, global_id 28230 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: _renew_subs 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbe90135690 con 0x7fbe9013e370 2024-09-15T20:33:40.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 client.28230.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: _renew_subs 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbe901d5ce0 con 0x7fbe9013e370 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe95e19640 1 librados: init done 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbe880039a0 con 0x7fbe9013e370 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbe88003df0 con 0x7fbe9013e370 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8e7fc640 4 set_mon_vals no callback set 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.586+0000 7fbe8cff9640 10 monclient: dump: 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:40.608 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe95e19640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe95e19640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe95e19640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe74005090 con 0x7fbe9013e370 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fbe88003f90 con 0x7fbe9013e370 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(73..73 src has 1..73) ==== 8319+0+0 (unknown 1040463339 0 0) 0x7fbe8804c5d0 con 0x7fbe9013e370 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe8cff9640 10 client.28230.objecter ms_dispatch 0x7fbe90000ba0 osd_map(73..73 src has 1..73) 2024-09-15T20:33:40.609 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe8cff9640 3 client.28230.objecter handle_osd_map got epochs [73,73] > 0 2024-09-15T20:33:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe8cff9640 3 client.28230.objecter handle_osd_map decoding full epoch 73 2024-09-15T20:33:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe8cff9640 20 client.28230.objecter dump_active .. 0 homeless 2024-09-15T20:33:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe8f7fe640 10 monclient: get_auth_request con 0x7fbe7003e330 auth_method 0 2024-09-15T20:33:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.590+0000 7fbe8cff9640 10 client.28230.objecter ms_handle_connect 0x7fbe7003e330 2024-09-15T20:33:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.594+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fbe88016d90 con 0x7fbe9013e370 2024-09-15T20:33:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.594+0000 7fbe8cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:40.610 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.594+0000 7fbe8cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:40.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.826+0000 7fbe95e19640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:33:40.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.826+0000 7fbe95e19640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:33:40.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.826+0000 7fbe95e19640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:40.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.826+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fbe74005c50 con 0x7fbe9013e370 2024-09-15T20:33:40.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.826+0000 7fbe8cff9640 1 -- 172.21.15.78:0/3351237804 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fbe88016690 con 0x7fbe9013e370 2024-09-15T20:33:40.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.826+0000 7fbe8cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:33:40.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.826+0000 7fbe8cff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:40.833 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.826+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.833 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.826+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=12,logm=61} crc {auth=4166290175,config=2586702516,health=3645530618,kv=199072906,logm=3122698576}) 2024-09-15T20:33:40.833 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.826+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=14,config=2,health=12,kv=11,logm=61} crc {auth=4166290175,config=2586702516,health=2784702441,kv=731607056,logm=1565279499}) 2024-09-15T20:33:40.834 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.826+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.834 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.826+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=14,config=2,health=11,kv=12,logm=61} crc {auth=4166290175,config=2586702516,health=3645530618,kv=199072906,logm=3122698576}) 2024-09-15T20:33:40.834 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.826+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=14,config=2,health=11,kv=12,logm=61} crc {auth=1524467002,config=2586702516,health=3801677455,kv=199072906,logm=1457581708}) 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 10 librados: watch_flush enter 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 10 librados: watch_flush exit 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 20 client.28230.objecter shutdown clearing up homeless session... 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 10 client.28230.objecter successfully canceled tick 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 >> v1:172.21.15.146:6812/435598642 conn(0x7fbe7003e330 legacy=0x7fbe700407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 10 monclient: shutdown 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 >> v1:172.21.15.78:6789/0 conn(0x7fbe9013e370 legacy=0x7fbe901d45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 shutdown_connections 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 >> 172.21.15.78:0/3351237804 conn(0x7fbe900aa650 msgr2=0x7fbe90140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:40.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 shutdown_connections 2024-09-15T20:33:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 1 -- 172.21.15.78:0/3351237804 wait complete. 2024-09-15T20:33:40.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:40.830+0000 7fbe95e19640 1 librados: shutdown 2024-09-15T20:33:40.836 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.836 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79108396}) 2024-09-15T20:33:40.836 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=657206741}) 2024-09-15T20:33:40.837 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.837 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=79108396}) 2024-09-15T20:33:40.837 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1415736025}) 2024-09-15T20:33:40.837 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.838 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402407648}) 2024-09-15T20:33:40.838 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=520671802}) 2024-09-15T20:33:40.838 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.838 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=402407648}) 2024-09-15T20:33:40.838 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.830+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4173604584}) 2024-09-15T20:33:40.840 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.834+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.840 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.834+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731389518}) 2024-09-15T20:33:40.840 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.834+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1571803942}) 2024-09-15T20:33:40.840 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.834+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.841 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.834+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=731389518}) 2024-09-15T20:33:40.841 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.834+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=8126905}) 2024-09-15T20:33:40.842 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.842 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221571825}) 2024-09-15T20:33:40.842 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3812966779}) 2024-09-15T20:33:40.842 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.842 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=221571825}) 2024-09-15T20:33:40.842 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=224204643}) 2024-09-15T20:33:40.843 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.843 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878885094}) 2024-09-15T20:33:40.844 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3811818620}) 2024-09-15T20:33:40.844 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.844 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2878885094}) 2024-09-15T20:33:40.844 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=517780090}) 2024-09-15T20:33:40.845 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.845 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279862549}) 2024-09-15T20:33:40.845 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372694233}) 2024-09-15T20:33:40.845 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.845 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2279862549}) 2024-09-15T20:33:40.846 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.838+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2914579579}) 2024-09-15T20:33:40.847 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.847 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943808677}) 2024-09-15T20:33:40.847 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=342588438}) 2024-09-15T20:33:40.847 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.847 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943808677}) 2024-09-15T20:33:40.847 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2299895985}) 2024-09-15T20:33:40.848 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.848 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370245178}) 2024-09-15T20:33:40.849 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=21027736}) 2024-09-15T20:33:40.849 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.849 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2370245178}) 2024-09-15T20:33:40.849 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.842+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1763822712}) 2024-09-15T20:33:40.850 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.850 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871089711}) 2024-09-15T20:33:40.850 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2131692055}) 2024-09-15T20:33:40.850 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.850 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=871089711}) 2024-09-15T20:33:40.850 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3728213615}) 2024-09-15T20:33:40.851 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:33:40.852 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.852 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512581456}) 2024-09-15T20:33:40.852 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1185958354}) 2024-09-15T20:33:40.852 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.852 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3512581456}) 2024-09-15T20:33:40.852 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.846+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1942063896}) 2024-09-15T20:33:40.853 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.853 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3} crc {logm=1277439245,mdsmap=215940802,mgr=3591229687,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=1112142692,mon_config_key=3126487007}) 2024-09-15T20:33:40.854 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=68,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=9} crc {logm=1978757621,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=406465119,mon_config_key=291070774}) 2024-09-15T20:33:40.854 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.854 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=2,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=3} crc {logm=1277439245,mdsmap=215940802,mgr=3591229687,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=1112142692,mon_config_key=3126487007}) 2024-09-15T20:33:40.854 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=71,mdsmap=2,mgr=8,mgr_command_descs=1,mgrstat=11,mon_config_key=7} crc {logm=1297851563,mdsmap=3140514009,mgr=2617012046,mgr_command_descs=2551042219,mgrstat=540080543,mon_config_key=3991244135}) 2024-09-15T20:33:40.856 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.856 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=13,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=60} crc {mon_config_key=390102175,monmap=3202364208,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=940291451,osd_snap=4025765695,osdmap=3987266825}) 2024-09-15T20:33:40.856 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=59} crc {mon_config_key=2726075306,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=940291451,osd_snap=553998847,osdmap=1730348132}) 2024-09-15T20:33:40.856 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.856 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=13,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=15,osdmap=60} crc {mon_config_key=390102175,monmap=3202364208,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=940291451,osd_snap=4025765695,osdmap=3987266825}) 2024-09-15T20:33:40.856 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.850+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=14,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=14,osdmap=60} crc {mon_config_key=2844817598,monmap=3202364208,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=940291452,osd_snap=3789555181,osdmap=2691045949}) 2024-09-15T20:33:40.858 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.854+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.858 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.854+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=2399083944}) 2024-09-15T20:33:40.858 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.854+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=61} crc {osdmap=649166067}) 2024-09-15T20:33:40.858 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.854+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:33:40.859 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.854+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=61} crc {osdmap=2399083944}) 2024-09-15T20:33:40.859 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:33:40.854+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=60} crc {osdmap=376905349}) 2024-09-15T20:33:41.853 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:33:41.853 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient: get_monmap_and_config 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient: build_initial_monmap 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient: monmap: 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:41.982227+0000 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:41.982227+0000 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:41.983 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:41.984 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:41.984 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:41.984 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 1 Processor -- start 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 1 -- start start 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient: init 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient(hunting): picked mon.noname-b con 0x7fee54134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient(hunting): picked mon.noname-a con 0x7fee5409f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient(hunting): picked mon.noname-c con 0x7fee5406e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:41.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee540a0470 con 0x7fee5409f5a0 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5409feb0 con 0x7fee5406e960 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5409e4a0 con 0x7fee54134bb0 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient(hunting): authenticate will time out at 1282.866678s 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee59255640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fee5409f5a0 0x7fee541399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36898/0 (socket says 172.21.15.78:36898) 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee59255640 1 -- 172.21.15.78:0/2235770538 learned_addr learned my addr 172.21.15.78:0/2235770538 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee59a56640 1 --1- 172.21.15.78:0/2235770538 >> v1:172.21.15.78:6790/0 conn(0x7fee54134bb0 0x7fee54134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43050/0 (socket says 172.21.15.78:43050) 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 587401362 0 0) 0x7fee44001120 con 0x7fee5409f5a0 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient(hunting): _negotiate 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:41.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient(hunting): my global_id is 28239 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee40003530 con 0x7fee5409f5a0 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 578453951 0 0) 0x7fee5409e4a0 con 0x7fee54134bb0 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient(hunting): _negotiate 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient(hunting): my global_id is 74135 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee44001120 con 0x7fee54134bb0 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2755305743 0 0) 0x7fee40003530 con 0x7fee5409f5a0 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 >> v1:172.21.15.78:6790/0 conn(0x7fee54134bb0 legacy=0x7fee54134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 >> v1:172.21.15.146:6789/0 conn(0x7fee5406e960 legacy=0x7fee5413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: _finish_hunting 0 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 monclient: found mon.noname-a 2024-09-15T20:33:41.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee540a0ff0 con 0x7fee5409f5a0 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: _finish_auth 0 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: _check_auth_tickets 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 5 monclient: authenticate success, global_id 28239 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fee44001870 con 0x7fee5409f5a0 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: dump: 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:41.993 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee44001a90 con 0x7fee5409f5a0 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 1 -- 172.21.15.78:0/2235770538 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fee440048b0 con 0x7fee5409f5a0 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee5bcba640 4 set_mon_vals no callback set 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.978+0000 7fee58a54640 10 monclient: dump: 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: shutdown 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:41.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- 172.21.15.78:0/2235770538 >> v1:172.21.15.78:6789/0 conn(0x7fee5409f5a0 legacy=0x7fee541399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- 172.21.15.78:0/2235770538 shutdown_connections 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- 172.21.15.78:0/2235770538 >> 172.21.15.78:0/2235770538 conn(0x7fee540a63c0 msgr2=0x7fee540a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- 172.21.15.78:0/2235770538 shutdown_connections 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- 172.21.15.78:0/2235770538 wait complete. 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: build_initial_monmap 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: monmap: 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:41.996 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 librados: starting msgr at 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 librados: starting objecter 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 Processor -- start 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- start start 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 librados: setting wanted keys 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 librados: calling monclient init 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: init 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient(hunting): picked mon.noname-a con 0x7fee5406e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:41.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient(hunting): picked mon.noname-c con 0x7fee5409f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient(hunting): picked mon.noname-b con 0x7fee54134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5411c780 con 0x7fee5406e960 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee541cb940 con 0x7fee54134bb0 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fee5413d9c0 con 0x7fee5409f5a0 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient(hunting): authenticate will time out at 1282.869922s 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee59a56640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fee5406e960 0x7fee541cab20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36908/0 (socket says 172.21.15.78:36908) 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee59a56640 1 -- 172.21.15.78:0/399476730 learned_addr learned my addr 172.21.15.78:0/399476730 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 client.?.objecter ms_handle_connect 0x7fee5409f5a0 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 client.?.objecter ms_handle_connect 0x7fee5406e960 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1044524088 0 0) 0x7fee5413d9c0 con 0x7fee5409f5a0 2024-09-15T20:33:41.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): my global_id is 74138 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee38004ef0 con 0x7fee5409f5a0 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2832192966 0 0) 0x7fee5411c780 con 0x7fee5406e960 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): my global_id is 28242 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee5413d9c0 con 0x7fee5406e960 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 client.?.objecter ms_handle_connect 0x7fee54134bb0 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4241090288 0 0) 0x7fee5413d9c0 con 0x7fee5406e960 2024-09-15T20:33:41.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee5411c780 con 0x7fee5406e960 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4116810349 0 0) 0x7fee38004ef0 con 0x7fee5409f5a0 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fee5413d9c0 con 0x7fee5409f5a0 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fee50001940 con 0x7fee5406e960 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): dump: 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.000 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fee440047f0 con 0x7fee5409f5a0 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): dump: 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.001 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4016902880 0 0) 0x7fee541cb940 con 0x7fee54134bb0 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient(hunting): my global_id is 22738 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fee38004ef0 con 0x7fee54134bb0 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 483690851 0 0) 0x7fee5411c780 con 0x7fee5406e960 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 >> v1:172.21.15.78:6790/0 conn(0x7fee5409f5a0 legacy=0x7fee541cb230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 >> v1:172.21.15.146:6789/0 conn(0x7fee54134bb0 legacy=0x7fee541d2d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 monclient: found mon.a 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee5413cc20 con 0x7fee5406e960 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 5 monclient: authenticate success, global_id 28242 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: _renew_subs 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fee5413e100 con 0x7fee5406e960 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 client.28242.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:42.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: _renew_subs 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fee5413bd20 con 0x7fee5406e960 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 librados: init done 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fee50001c80 con 0x7fee5406e960 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fee50004c40 con 0x7fee5406e960 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 10 monclient: dump: 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.004 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4bfff640 4 set_mon_vals no callback set 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee3c005090 con 0x7fee5406e960 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.982+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fee50004de0 con 0x7fee5406e960 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.986+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(74..74 src has 1..74) ==== 7466+0+0 (unknown 584100179 0 0) 0x7fee5004c080 con 0x7fee5406e960 2024-09-15T20:33:42.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.986+0000 7fee4a7fc640 10 client.28242.objecter ms_dispatch 0x7fee54000ba0 osd_map(74..74 src has 1..74) 2024-09-15T20:33:42.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.986+0000 7fee4a7fc640 3 client.28242.objecter handle_osd_map got epochs [74,74] > 0 2024-09-15T20:33:42.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.986+0000 7fee4a7fc640 3 client.28242.objecter handle_osd_map decoding full epoch 74 2024-09-15T20:33:42.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.986+0000 7fee4a7fc640 20 client.28242.objecter dump_active .. 0 homeless 2024-09-15T20:33:42.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.986+0000 7fee59255640 10 monclient: get_auth_request con 0x7fee38042640 auth_method 0 2024-09-15T20:33:42.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.986+0000 7fee4a7fc640 10 client.28242.objecter ms_handle_connect 0x7fee38042640 2024-09-15T20:33:42.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.990+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fee50016ae0 con 0x7fee5406e960 2024-09-15T20:33:42.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.990+0000 7fee4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:41.990+0000 7fee4a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:42.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.222+0000 7fee5bcba640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:33:42.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.222+0000 7fee5bcba640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:42.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.222+0000 7fee5bcba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.222+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fee3c005380 con 0x7fee5406e960 2024-09-15T20:33:42.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.222+0000 7fee4a7fc640 1 -- 172.21.15.78:0/399476730 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3980384259) 0x7fee500163e0 con 0x7fee5406e960 2024-09-15T20:33:42.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.222+0000 7fee4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:42.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.222+0000 7fee4a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:42.228 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 10 librados: watch_flush enter 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 10 librados: watch_flush exit 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 20 client.28242.objecter shutdown clearing up homeless session... 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 10 client.28242.objecter successfully canceled tick 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 >> v1:172.21.15.146:6812/435598642 conn(0x7fee38042640 legacy=0x7fee38044b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 10 monclient: shutdown 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 >> v1:172.21.15.78:6789/0 conn(0x7fee5406e960 legacy=0x7fee541cab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 shutdown_connections 2024-09-15T20:33:42.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 >> 172.21.15.78:0/399476730 conn(0x7fee540a63c0 msgr2=0x7fee540732a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:42.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 shutdown_connections 2024-09-15T20:33:42.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 1 -- 172.21.15.78:0/399476730 wait complete. 2024-09-15T20:33:42.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.226+0000 7fee5bcba640 1 librados: shutdown 2024-09-15T20:33:42.246 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:33:42.246 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:33:42.246 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:33:42.246 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: get_monmap_and_config 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: build_initial_monmap 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: monmap: 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:42.361248+0000 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:42.361248+0000 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:42.362 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:42.363 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:42.363 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 Processor -- start 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- start start 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: init 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient(hunting): picked mon.noname-b con 0x7f1d7c09e590 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient(hunting): picked mon.noname-a con 0x7f1d7c09e970 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient(hunting): picked mon.noname-c con 0x7f1d7c09edd0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d7c0a4790 con 0x7f1d7c09e970 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d7c0a5a30 con 0x7f1d7c09edd0 2024-09-15T20:33:42.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d7c06e3d0 con 0x7f1d7c09e590 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient(hunting): authenticate will time out at 1283.245742s 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7affd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1d7c09e970 0x7f1d7c13ccf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36924/0 (socket says 172.21.15.78:36924) 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7affd640 1 -- 172.21.15.78:0/735554432 learned_addr learned my addr 172.21.15.78:0/735554432 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4228603760 0 0) 0x7f1d7c0a4790 con 0x7f1d7c09e970 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient(hunting): my global_id is 28245 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d64003530 con 0x7f1d7c09e970 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1631339100 0 0) 0x7f1d7c06e3d0 con 0x7f1d7c09e590 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient(hunting): my global_id is 74141 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d7c0a4790 con 0x7f1d7c09e590 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 149721848 0 0) 0x7f1d7c0a4790 con 0x7f1d7c09e590 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 >> v1:172.21.15.146:6789/0 conn(0x7f1d7c09edd0 legacy=0x7f1d7c140420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 >> v1:172.21.15.78:6789/0 conn(0x7f1d7c09e970 legacy=0x7f1d7c13ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 monclient: found mon.noname-b 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c06e6b0 con 0x7f1d7c09e590 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1d70001680 con 0x7f1d7c09e590 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: dump: 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.373 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 5 monclient: authenticate success, global_id 74141 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1d70001820 con 0x7f1d7c09e590 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 1 -- 172.21.15.78:0/735554432 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1d70001c30 con 0x7f1d7c09e590 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d7a7fc640 10 monclient: dump: 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.374 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 4 set_mon_vals no callback set 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: shutdown 2024-09-15T20:33:42.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- 172.21.15.78:0/735554432 >> v1:172.21.15.78:6790/0 conn(0x7f1d7c09e590 legacy=0x7f1d7c1349b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- 172.21.15.78:0/735554432 shutdown_connections 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- 172.21.15.78:0/735554432 >> 172.21.15.78:0/735554432 conn(0x7f1d7c0a4c20 msgr2=0x7f1d7c0a5030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- 172.21.15.78:0/735554432 shutdown_connections 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 1 -- 172.21.15.78:0/735554432 wait complete. 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.358+0000 7f1d81bff640 10 monclient: build_initial_monmap 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient: monmap: 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:42.376 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 librados: starting msgr at 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 librados: starting objecter 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 Processor -- start 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 -- start start 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 librados: setting wanted keys 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 librados: calling monclient init 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient: init 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:42.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient(hunting): picked mon.noname-a con 0x7f1d7c09e590 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient(hunting): picked mon.noname-b con 0x7f1d7c09e970 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient(hunting): picked mon.noname-c con 0x7f1d7c09edd0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d7c1303e0 con 0x7f1d7c09e590 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d7c1306c0 con 0x7f1d7c09e970 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1d7c12e000 con 0x7f1d7c09edd0 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient(hunting): authenticate will time out at 1283.248950s 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d7b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1d7c09e590 0x7f1d7c12d1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36928/0 (socket says 172.21.15.78:36928) 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d7b7fe640 1 -- 172.21.15.78:0/3516252811 learned_addr learned my addr 172.21.15.78:0/3516252811 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.?.objecter ms_handle_connect 0x7f1d7c09edd0 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.?.objecter ms_handle_connect 0x7f1d7c09e590 2024-09-15T20:33:42.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1421146704 0 0) 0x7f1d7c12e000 con 0x7f1d7c09edd0 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): my global_id is 74144 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d68004ec0 con 0x7f1d7c09edd0 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2280857589 0 0) 0x7f1d7c1303e0 con 0x7f1d7c09e590 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): my global_id is 28248 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d7c12e000 con 0x7f1d7c09e590 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.?.objecter ms_handle_connect 0x7f1d7c09e970 2024-09-15T20:33:42.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 63773413 0 0) 0x7f1d68004ec0 con 0x7f1d7c09edd0 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d7c1303e0 con 0x7f1d7c09edd0 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1d74001c40 con 0x7f1d7c09edd0 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): dump: 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.380 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2148153284 0 0) 0x7f1d7c12e000 con 0x7f1d7c09e590 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d68004ec0 con 0x7f1d7c09e590 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 134393637 0 0) 0x7f1d7c1306c0 con 0x7f1d7c09e970 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): my global_id is 22744 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d7c12e000 con 0x7f1d7c09e970 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1d70001680 con 0x7f1d7c09e590 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): dump: 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.382 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2774004102 0 0) 0x7f1d7c1303e0 con 0x7f1d7c09edd0 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 >> v1:172.21.15.146:6789/0 conn(0x7f1d7c09e970 legacy=0x7f1d7c12d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 >> v1:172.21.15.78:6789/0 conn(0x7f1d7c09e590 legacy=0x7f1d7c12d1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 monclient: found mon.c 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d7c12e360 con 0x7f1d7c09edd0 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 5 monclient: authenticate success, global_id 74144 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient: _renew_subs 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1d7c12fe20 con 0x7f1d7c09edd0 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1d74003980 con 0x7f1d7c09edd0 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 client.74144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient: _renew_subs 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1d74003cd0 con 0x7f1d7c09edd0 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1d7c12f090 con 0x7f1d7c09edd0 2024-09-15T20:33:42.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 librados: init done 2024-09-15T20:33:42.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:42.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d79ffb640 4 set_mon_vals no callback set 2024-09-15T20:33:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: dump: 2024-09-15T20:33:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.388 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1d7c0008e0 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.74144.objecter ms_handle_connect 0x7f1d7c0a1e50 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.74144.objecter resend_mon_ops 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1d7400fd40 con 0x7f1d7c09edd0 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(74..74 src has 1..74) ==== 7466+0+0 (unknown 584100179 0 0) 0x7f1d7404b040 con 0x7f1d7c09edd0 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.74144.objecter ms_dispatch 0x7f1d7c000ba0 osd_map(74..74 src has 1..74) 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 3 client.74144.objecter handle_osd_map got epochs [74,74] > 0 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 3 client.74144.objecter handle_osd_map decoding full epoch 74 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 20 client.74144.objecter dump_active .. 0 homeless 2024-09-15T20:33:42.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 895406801 0 0) 0x7f1d7c0008e0 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient(hunting): my global_id is 28251 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d7c1303e0 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d7affd640 10 monclient: get_auth_request con 0x7f1d6803e210 auth_method 0 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 38704323 0 0) 0x7f1d7c1303e0 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d7c0008e0 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1d70003b10 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 monclient: dump: 2024-09-15T20:33:42.390 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3381829998 0 0) 0x7f1d7c0008e0 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1d7c0a2230 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.362+0000 7f1d63fff640 10 client.74144.objecter ms_handle_connect 0x7f1d6803e210 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.366+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1d70005850 con 0x7f1d7c0a1e50 2024-09-15T20:33:42.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.366+0000 7f1d63fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.366+0000 7f1d63fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.366+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 >> v1:172.21.15.78:6789/0 conn(0x7f1d7c0a1e50 legacy=0x7f1d7c0a67a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.366+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 >> v1:172.21.15.78:6789/0 conn(0x7f1d7c0a1e50 legacy=0x7f1d7c0a67a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:42.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:33:42.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1d7c141b40 con 0x7f1d7c02e460 2024-09-15T20:33:42.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 client.74144.objecter ms_handle_connect 0x7f1d7c02e460 2024-09-15T20:33:42.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 client.74144.objecter resend_mon_ops 2024-09-15T20:33:42.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 755838422 0 0) 0x7f1d7c141b40 con 0x7f1d7c02e460 2024-09-15T20:33:42.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient(hunting): my global_id is 28254 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d7c0008e0 con 0x7f1d7c02e460 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1092318841 0 0) 0x7f1d7c0008e0 con 0x7f1d7c02e460 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d7c141b40 con 0x7f1d7c02e460 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1d70003e80 con 0x7f1d7c02e460 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient: dump: 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.394 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 314562496 0 0) 0x7f1d7c141b40 con 0x7f1d7c02e460 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1d7c02e840 con 0x7f1d7c02e460 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4140 (unknown 0 0 424362901) 0x7f1d70003eb0 con 0x7f1d7c02e460 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:42.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 >> v1:172.21.15.78:6789/0 conn(0x7f1d7c02e460 legacy=0x7f1d7c0a6eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d63fff640 1 -- 172.21.15.78:0/3516252811 >> v1:172.21.15.78:6789/0 conn(0x7f1d7c02e460 legacy=0x7f1d7c0a6eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 10, 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 134921, 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:33:42.396 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.397 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.398 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:42.399 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.400 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.401 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:42.402 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.403 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:33:42.404 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.405 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:33:42.406 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.407 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:33:42.407 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:33:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 10 librados: watch_flush enter 2024-09-15T20:33:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 10 librados: watch_flush exit 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 20 client.74144.objecter shutdown clearing up homeless session... 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 10 client.74144.objecter successfully canceled tick 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 >> v1:172.21.15.146:6812/435598642 conn(0x7f1d6803e210 legacy=0x7f1d680406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 10 monclient: shutdown 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 >> v1:172.21.15.78:6790/0 conn(0x7f1d7c09edd0 legacy=0x7f1d7c1d4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d7bfff640 1 -- 172.21.15.78:0/3516252811 reap_dead start 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d7bfff640 1 -- 172.21.15.78:0/3516252811 reap_dead start 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 shutdown_connections 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 >> 172.21.15.78:0/3516252811 conn(0x7f1d7c0a4c20 msgr2=0x7f1d7c1375d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 shutdown_connections 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 1 -- 172.21.15.78:0/3516252811 wait complete. 2024-09-15T20:33:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.370+0000 7f1d81bff640 1 librados: shutdown 2024-09-15T20:33:42.410 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:33:42.410 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:33:42.410 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient: get_monmap_and_config 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient: build_initial_monmap 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient: monmap: 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:42.522260+0000 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:42.522260+0000 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.524 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:42.525 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 1 Processor -- start 2024-09-15T20:33:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 1 -- start start 2024-09-15T20:33:42.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient: init 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f89fc0a3580 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f89fc130720 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f89fc131260 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89fc09f3d0 con 0x7f89fc130720 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89fc0a1ff0 con 0x7f89fc131260 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89fc1357d0 con 0x7f89fc0a3580 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a03a0e640 10 monclient(hunting): authenticate will time out at 1283.407190s 2024-09-15T20:33:42.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a017aa640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f89fc0a3580 0x7f89fc09fcd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43088/0 (socket says 172.21.15.78:43088) 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f8a017aa640 1 -- 172.21.15.78:0/867572834 learned_addr learned my addr 172.21.15.78:0/867572834 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3040614356 0 0) 0x7f89fc1357d0 con 0x7f89fc0a3580 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f89f3fff640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f89f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f89f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f89f3fff640 10 monclient(hunting): my global_id is 74147 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.518+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89e8003530 con 0x7f89fc0a3580 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 913827988 0 0) 0x7f89e8003530 con 0x7f89fc0a3580 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 >> v1:172.21.15.146:6789/0 conn(0x7f89fc131260 legacy=0x7f89fc1350c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 >> v1:172.21.15.78:6789/0 conn(0x7f89fc130720 legacy=0x7f89fc130b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 1 monclient: found mon.noname-b 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc1359a0 con 0x7f89fc0a3580 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f89f80016a0 con 0x7f89fc0a3580 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: dump: 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.529 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.530 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f8a03a0e640 5 monclient: authenticate success, global_id 74147 2024-09-15T20:33:42.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f8a03a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89f8001840 con 0x7f89fc0a3580 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 1 -- 172.21.15.78:0/867572834 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f89f8001cd0 con 0x7f89fc0a3580 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f89f3fff640 10 monclient: dump: 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.531 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f8a03a0e640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f8a03a0e640 4 set_mon_vals no callback set 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.522+0000 7f8a03a0e640 10 monclient: shutdown 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- 172.21.15.78:0/867572834 >> v1:172.21.15.78:6790/0 conn(0x7f89fc0a3580 legacy=0x7f89fc09fcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- 172.21.15.78:0/867572834 shutdown_connections 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- 172.21.15.78:0/867572834 >> 172.21.15.78:0/867572834 conn(0x7f89fc0a2a90 msgr2=0x7f89fc0a2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:42.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- 172.21.15.78:0/867572834 shutdown_connections 2024-09-15T20:33:42.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- 172.21.15.78:0/867572834 wait complete. 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient: build_initial_monmap 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient: monmap: 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:42.536 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 librados: starting msgr at 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 librados: starting objecter 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 Processor -- start 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- start start 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 librados: setting wanted keys 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 librados: calling monclient init 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient: init 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f89fc0a3580 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:42.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f89fc130720 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f89fc131260 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89fc0700f0 con 0x7f89fc130720 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89fc145b70 con 0x7f89fc131260 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89fc0a3d90 con 0x7f89fc0a3580 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a03a0e640 10 monclient(hunting): authenticate will time out at 1283.415163s 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a017aa640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f89fc0a3580 0x7f89fc144d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43098/0 (socket says 172.21.15.78:43098) 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f8a017aa640 1 -- 172.21.15.78:0/2542091433 learned_addr learned my addr 172.21.15.78:0/2542091433 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 client.?.objecter ms_handle_connect 0x7f89fc0a3580 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 client.?.objecter ms_handle_connect 0x7f89fc130720 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507891847 0 0) 0x7f89fc0a3d90 con 0x7f89fc0a3580 2024-09-15T20:33:42.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): my global_id is 74150 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89e0004ec0 con 0x7f89fc0a3580 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2366993565 0 0) 0x7f89fc0700f0 con 0x7f89fc130720 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): my global_id is 28257 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89fc0a3d90 con 0x7f89fc130720 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2994474921 0 0) 0x7f89fc0a3d90 con 0x7f89fc130720 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89fc0700f0 con 0x7f89fc130720 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 626105981 0 0) 0x7f89e0004ec0 con 0x7f89fc0a3580 2024-09-15T20:33:42.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89fc0a3d90 con 0x7f89fc0a3580 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f89ec0018b0 con 0x7f89fc130720 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.526+0000 7f89f1ffb640 10 monclient(hunting): dump: 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.540 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f89f8001b90 con 0x7f89fc0a3580 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient(hunting): dump: 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.541 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4012874679 0 0) 0x7f89fc0700f0 con 0x7f89fc130720 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 >> v1:172.21.15.78:6790/0 conn(0x7f89fc0a3580 legacy=0x7f89fc144d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 >> v1:172.21.15.146:6789/0 conn(0x7f89fc131260 legacy=0x7f89fc1d9620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 monclient: found mon.a 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89fc0a4130 con 0x7f89fc130720 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 5 monclient: authenticate success, global_id 28257 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 10 monclient: _renew_subs 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f89fc06fe10 con 0x7f89fc130720 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89ec003600 con 0x7f89fc130720 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f89ec003990 con 0x7f89fc130720 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient: dump: 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.543 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 10 client.28257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 10 monclient: _renew_subs 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f89fc073c80 con 0x7f89fc130720 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 1 librados: init done 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f37fe640 4 set_mon_vals no callback set 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f89ec003bb0 con 0x7f89fc130720 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 7111+0+0 (unknown 4011571972 0 0) 0x7f89ec048910 con 0x7f89fc130720 2024-09-15T20:33:42.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 client.28257.objecter ms_dispatch 0x7f89fc000ba0 osd_map(75..75 src has 1..75) 2024-09-15T20:33:42.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 3 client.28257.objecter handle_osd_map got epochs [75,75] > 0 2024-09-15T20:33:42.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 3 client.28257.objecter handle_osd_map decoding full epoch 75 2024-09-15T20:33:42.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 20 client.28257.objecter dump_active .. 0 homeless 2024-09-15T20:33:42.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a017aa640 10 monclient: get_auth_request con 0x7f89e003e330 auth_method 0 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f89dc0065d0 con 0x7f89dc002870 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 client.28257.objecter ms_handle_connect 0x7f89dc002870 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 client.28257.objecter resend_mon_ops 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 client.28257.objecter ms_handle_connect 0x7f89e003e330 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3372720389 0 0) 0x7f89dc0065d0 con 0x7f89dc002870 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 10 monclient(hunting): my global_id is 74153 2024-09-15T20:33:42.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.530+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89fc0700f0 con 0x7f89dc002870 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2101434871 0 0) 0x7f89fc0700f0 con 0x7f89dc002870 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89dc0065d0 con 0x7f89dc002870 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f89f4001a50 con 0x7f89dc002870 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 10 monclient: dump: 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.549 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2327863147 0 0) 0x7f89dc0065d0 con 0x7f89dc002870 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f89dc00a170 con 0x7f89dc002870 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f89f4002040 con 0x7f89dc002870 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 >> v1:172.21.15.78:6790/0 conn(0x7f89dc002870 legacy=0x7f89dc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.534+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 >> v1:172.21.15.78:6790/0 conn(0x7f89dc002870 legacy=0x7f89dc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.538+0000 7f8a03a0e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.538+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f89dc009e80 con 0x7f89dc009140 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 client.28257.objecter ms_handle_connect 0x7f89dc009140 2024-09-15T20:33:42.550 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 client.28257.objecter resend_mon_ops 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3704470304 0 0) 0x7f89dc009e80 con 0x7f89dc009140 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient(hunting): my global_id is 74156 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89dc0065d0 con 0x7f89dc009140 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1464118348 0 0) 0x7f89dc0065d0 con 0x7f89dc009140 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f89dc009e80 con 0x7f89dc009140 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f89f4003d90 con 0x7f89dc009140 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient: dump: 2024-09-15T20:33:42.551 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2343733908 0 0) 0x7f89dc009e80 con 0x7f89dc009140 2024-09-15T20:33:42.552 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f89dc00c420 con 0x7f89dc009140 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3969 (unknown 0 0 4194008516) 0x7f89f40049d0 con 0x7f89dc009140 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 >> v1:172.21.15.78:6790/0 conn(0x7f89dc009140 legacy=0x7f89dc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f89f1ffb640 1 -- 172.21.15.78:0/2542091433 >> v1:172.21.15.78:6790/0 conn(0x7f89dc009140 legacy=0x7f89dc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:42.553 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 10, 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 2939, 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.554 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.555 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:33:42.556 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:33:42.557 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.558 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.559 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.560 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.561 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.562 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:33:42.563 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.564 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.564 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.564 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:42.564 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.564 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.564 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.564 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:33:42.564 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 10 librados: watch_flush enter 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 10 librados: watch_flush exit 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 20 client.28257.objecter shutdown clearing up homeless session... 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 10 client.28257.objecter successfully canceled tick 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 >> v1:172.21.15.146:6812/435598642 conn(0x7f89e003e330 legacy=0x7f89e00407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 10 monclient: shutdown 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 >> v1:172.21.15.78:6789/0 conn(0x7f89fc130720 legacy=0x7f89fc145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a01fab640 1 -- 172.21.15.78:0/2542091433 reap_dead start 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 shutdown_connections 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 >> 172.21.15.78:0/2542091433 conn(0x7f89fc0a2a90 msgr2=0x7f89fc12f090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 shutdown_connections 2024-09-15T20:33:42.566 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 1 -- 172.21.15.78:0/2542091433 wait complete. 2024-09-15T20:33:42.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.542+0000 7f8a03a0e640 1 librados: shutdown 2024-09-15T20:33:42.567 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:33:42.567 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:33:42.567 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:33:42.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient: get_monmap_and_config 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient: build_initial_monmap 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient: monmap: 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:42.677948+0000 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:42.677948+0000 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:42.679 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 1 Processor -- start 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 1 -- start start 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient: init 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient(hunting): picked mon.noname-b con 0x7f87f8074630 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient(hunting): picked mon.noname-a con 0x7f87f809d950 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient(hunting): picked mon.noname-c con 0x7f87f8074a10 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:42.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f8134bc0 con 0x7f87f809d950 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f809f900 con 0x7f87f8074a10 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f809fbe0 con 0x7f87f8074630 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient(hunting): authenticate will time out at 1283.562547s 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fcbe5640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f87f8074630 0x7f87f8133da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43116/0 (socket says 172.21.15.78:43116) 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fcbe5640 1 -- 172.21.15.78:0/2173893374 learned_addr learned my addr 172.21.15.78:0/2173893374 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f7fff640 1 --1- 172.21.15.78:0/2173893374 >> v1:172.21.15.78:6789/0 conn(0x7f87f809d950 0x7f87f81344b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36954/0 (socket says 172.21.15.78:36954) 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3525480085 0 0) 0x7f87f8134bc0 con 0x7f87f809d950 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient(hunting): my global_id is 28266 2024-09-15T20:33:42.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87e0004640 con 0x7f87f809d950 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2117274850 0 0) 0x7f87f809fbe0 con 0x7f87f8074630 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient(hunting): my global_id is 74165 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f8134bc0 con 0x7f87f8074630 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 648034536 0 0) 0x7f87e0004640 con 0x7f87f809d950 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 >> v1:172.21.15.78:6790/0 conn(0x7f87f8074630 legacy=0x7f87f8133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 >> v1:172.21.15.146:6789/0 conn(0x7f87f8074a10 legacy=0x7f87f813d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 monclient: found mon.noname-a 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f809fec0 con 0x7f87f809d950 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87ec0016a0 con 0x7f87f809d950 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: dump: 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.691 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 5 monclient: authenticate success, global_id 28266 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87ec001840 con 0x7f87f809d950 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87fee49640 4 set_mon_vals no callback set 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 1 -- 172.21.15.78:0/2173893374 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87ec001c50 con 0x7f87f809d950 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.674+0000 7f87f77fe640 10 monclient: dump: 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.693 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: shutdown 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- 172.21.15.78:0/2173893374 >> v1:172.21.15.78:6789/0 conn(0x7f87f809d950 legacy=0x7f87f81344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- 172.21.15.78:0/2173893374 shutdown_connections 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- 172.21.15.78:0/2173893374 >> 172.21.15.78:0/2173893374 conn(0x7f87f80a63c0 msgr2=0x7f87f80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- 172.21.15.78:0/2173893374 shutdown_connections 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- 172.21.15.78:0/2173893374 wait complete. 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: build_initial_monmap 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: monmap: 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:42.694 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 librados: starting msgr at 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 librados: starting objecter 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 Processor -- start 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- start start 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 librados: setting wanted keys 2024-09-15T20:33:42.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 librados: calling monclient init 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: init 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient(hunting): picked mon.noname-c con 0x7f87f8074630 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient(hunting): picked mon.noname-b con 0x7f87f8074a10 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient(hunting): picked mon.noname-a con 0x7f87f809d950 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f811e070 con 0x7f87f809d950 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f811e350 con 0x7f87f8074a10 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f80a5450 con 0x7f87f8074630 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient(hunting): authenticate will time out at 1283.565899s 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fcbe5640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f87f8074630 0x7f87f812dc20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43132/0 (socket says 172.21.15.78:43132) 2024-09-15T20:33:42.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fcbe5640 1 -- 172.21.15.78:0/2077173587 learned_addr learned my addr 172.21.15.78:0/2077173587 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 client.?.objecter ms_handle_connect 0x7f87f8074630 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 client.?.objecter ms_handle_connect 0x7f87f809d950 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3698455907 0 0) 0x7f87f80a5450 con 0x7f87f8074630 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): my global_id is 74168 2024-09-15T20:33:42.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87e4004ec0 con 0x7f87f8074630 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1507967776 0 0) 0x7f87f811e070 con 0x7f87f809d950 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): my global_id is 28269 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f80a5450 con 0x7f87f809d950 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 client.?.objecter ms_handle_connect 0x7f87f8074a10 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 933496173 0 0) 0x7f87e4004ec0 con 0x7f87f8074630 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87f811e070 con 0x7f87f8074630 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87e8001a20 con 0x7f87f8074630 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): dump: 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:42.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1258772022 0 0) 0x7f87f80a5450 con 0x7f87f809d950 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87e4004ec0 con 0x7f87f809d950 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87f0001c00 con 0x7f87f809d950 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): dump: 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.700 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 649504567 0 0) 0x7f87f811e350 con 0x7f87f8074a10 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient(hunting): my global_id is 22756 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f80a5450 con 0x7f87f8074a10 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1788235933 0 0) 0x7f87f811e070 con 0x7f87f8074630 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 >> v1:172.21.15.146:6789/0 conn(0x7f87f8074a10 legacy=0x7f87f81d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 >> v1:172.21.15.78:6789/0 conn(0x7f87f809d950 legacy=0x7f87f81d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 monclient: found mon.c 2024-09-15T20:33:42.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f81baad0 con 0x7f87f8074630 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 5 monclient: authenticate success, global_id 74168 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: _renew_subs 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87f809d180 con 0x7f87f8074630 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 client.74168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: _renew_subs 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f87f812e330 con 0x7f87f8074630 2024-09-15T20:33:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 librados: init done 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87e8001630 con 0x7f87f8074630 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87e8004c40 con 0x7f87f8074630 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f67fc640 4 set_mon_vals no callback set 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 10 monclient: dump: 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f87d80065d0 con 0x7f87d8002870 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.678+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f87e8010cb0 con 0x7f87f8074630 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(75..75 src has 1..75) ==== 7111+0+0 (unknown 4011571972 0 0) 0x7f87e804beb0 con 0x7f87f8074630 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 client.74168.objecter ms_dispatch 0x7f87f8000ba0 osd_map(75..75 src has 1..75) 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 3 client.74168.objecter handle_osd_map got epochs [75,75] > 0 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 3 client.74168.objecter handle_osd_map decoding full epoch 75 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 20 client.74168.objecter dump_active .. 0 homeless 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 client.74168.objecter ms_handle_connect 0x7f87d8002870 2024-09-15T20:33:42.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 client.74168.objecter resend_mon_ops 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87fd3e6640 10 monclient: get_auth_request con 0x7f87e403e260 auth_method 0 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1580896164 0 0) 0x7f87d80065d0 con 0x7f87d8002870 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient(hunting): my global_id is 22759 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f811e070 con 0x7f87d8002870 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 client.74168.objecter ms_handle_connect 0x7f87e403e260 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2549995624 0 0) 0x7f87f811e070 con 0x7f87d8002870 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87d80065d0 con 0x7f87d8002870 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87ec001790 con 0x7f87d8002870 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient: dump: 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1995515833 0 0) 0x7f87d80065d0 con 0x7f87d8002870 2024-09-15T20:33:42.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87d800a170 con 0x7f87d8002870 2024-09-15T20:33:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f87ec001ab0 con 0x7f87d8002870 2024-09-15T20:33:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 >> v1:172.21.15.146:6789/0 conn(0x7f87d8002870 legacy=0x7f87d8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.682+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 >> v1:172.21.15.146:6789/0 conn(0x7f87d8002870 legacy=0x7f87d8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f87d8009e60 con 0x7f87d8009120 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 client.74168.objecter ms_handle_connect 0x7f87d8009120 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 client.74168.objecter resend_mon_ops 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1587531990 0 0) 0x7f87d8009e60 con 0x7f87d8009120 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient(hunting): my global_id is 22762 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87d80065d0 con 0x7f87d8009120 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4151502781 0 0) 0x7f87d80065d0 con 0x7f87d8009120 2024-09-15T20:33:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87d8009e60 con 0x7f87d8009120 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87ec004c20 con 0x7f87d8009120 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient: dump: 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4264093421 0 0) 0x7f87d8009e60 con 0x7f87d8009120 2024-09-15T20:33:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f87d800c400 con 0x7f87d8009120 2024-09-15T20:33:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4306 (unknown 0 0 544360939) 0x7f87ec004a80 con 0x7f87d8009120 2024-09-15T20:33:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:33:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 >> v1:172.21.15.146:6789/0 conn(0x7f87d8009120 legacy=0x7f87d80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87f57fa640 1 -- 172.21.15.78:0/2077173587 >> v1:172.21.15.146:6789/0 conn(0x7f87d8009120 legacy=0x7f87d80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:42.711 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:33:42.711 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 10, 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 135198, 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 3, 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:33:42.712 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.713 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.714 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:42.715 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:42.716 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:33:42.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.719 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.720 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:33:42.721 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.722 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 10 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.723 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:33:42.724 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:33:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 10 librados: watch_flush enter 2024-09-15T20:33:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 10 librados: watch_flush exit 2024-09-15T20:33:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 20 client.74168.objecter shutdown clearing up homeless session... 2024-09-15T20:33:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 10 client.74168.objecter successfully canceled tick 2024-09-15T20:33:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 >> v1:172.21.15.146:6812/435598642 conn(0x7f87e403e260 legacy=0x7f87e4040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 10 monclient: shutdown 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 >> v1:172.21.15.78:6790/0 conn(0x7f87f8074630 legacy=0x7f87f812dc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fd3e6640 1 -- 172.21.15.78:0/2077173587 reap_dead start 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fd3e6640 1 -- 172.21.15.78:0/2077173587 reap_dead start 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 shutdown_connections 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 >> 172.21.15.78:0/2077173587 conn(0x7f87f80a63c0 msgr2=0x7f87f80a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 shutdown_connections 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 1 -- 172.21.15.78:0/2077173587 wait complete. 2024-09-15T20:33:42.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.690+0000 7f87fee49640 1 librados: shutdown 2024-09-15T20:33:42.727 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:33:42.728 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T20:33:42.728 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T20:33:42.728 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T20:33:42.728 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T20:33:42.728 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T20:33:42.728 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:33:42.738 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T20:33:42.738 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:33:42.739 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient: get_monmap_and_config 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient: build_initial_monmap 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient: monmap: 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:42.894272+0000 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:42.894272+0000 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:42.895 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:42.896 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 1 Processor -- start 2024-09-15T20:33:42.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 1 -- start start 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient: init 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient(hunting): picked mon.noname-b con 0x7f58f8135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient(hunting): picked mon.noname-c con 0x7f58f80a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient(hunting): picked mon.noname-a con 0x7f58f813c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f80745e0 con 0x7f58f813c220 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f806ecf0 con 0x7f58f80a24e0 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f806efd0 con 0x7f58f8135b60 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient(hunting): authenticate will time out at 1283.778850s 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fe514640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f58f8135b60 legacy=0x7f58f8135f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:42.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fed15640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f58f813c220 0x7f58f813e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36972/0 (socket says 172.21.15.78:36972) 2024-09-15T20:33:42.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fed15640 1 -- 172.21.15.78:0/2910020452 learned_addr learned my addr 172.21.15.78:0/2910020452 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:42.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3650667293 0 0) 0x7f58f80745e0 con 0x7f58f813c220 2024-09-15T20:33:42.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient(hunting): my global_id is 28272 2024-09-15T20:33:42.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58e4003530 con 0x7f58f813c220 2024-09-15T20:33:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2243885474 0 0) 0x7f58f806ecf0 con 0x7f58f80a24e0 2024-09-15T20:33:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient(hunting): my global_id is 22765 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58f80745e0 con 0x7f58f80a24e0 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 847546910 0 0) 0x7f58e4003530 con 0x7f58f813c220 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 >> v1:172.21.15.78:6790/0 conn(0x7f58f8135b60 legacy=0x7f58f8135f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 >> v1:172.21.15.146:6789/0 conn(0x7f58f80a24e0 legacy=0x7f58f813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 monclient: found mon.noname-a 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f8136650 con 0x7f58f813c220 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 5 monclient: authenticate success, global_id 28272 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f58f0001a20 con 0x7f58f813c220 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: dump: 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.905 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58f0001c40 con 0x7f58f813c220 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f5900778640 4 set_mon_vals no callback set 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 1 -- 172.21.15.78:0/2910020452 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f58f0002010 con 0x7f58f813c220 2024-09-15T20:33:42.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.890+0000 7f58fd512640 10 monclient: dump: 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: shutdown 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:42.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- 172.21.15.78:0/2910020452 >> v1:172.21.15.78:6789/0 conn(0x7f58f813c220 legacy=0x7f58f813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- 172.21.15.78:0/2910020452 shutdown_connections 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- 172.21.15.78:0/2910020452 >> 172.21.15.78:0/2910020452 conn(0x7f58f80a63c0 msgr2=0x7f58f80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- 172.21.15.78:0/2910020452 shutdown_connections 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- 172.21.15.78:0/2910020452 wait complete. 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: build_initial_monmap 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: monmap: 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.908 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 librados: starting msgr at 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 librados: starting objecter 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 Processor -- start 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- start start 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 librados: setting wanted keys 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 librados: calling monclient init 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: init 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:33:42.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient(hunting): picked mon.noname-a con 0x7f58f80a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient(hunting): picked mon.noname-b con 0x7f58f8135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient(hunting): picked mon.noname-c con 0x7f58f813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f809d180 con 0x7f58f80a24e0 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f81384f0 con 0x7f58f8135b60 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f58f8136fd0 con 0x7f58f813c220 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient(hunting): authenticate will time out at 1283.782223s 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58fed15640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f58f813c220 legacy=0x7f58f8072a80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58fe514640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f58f80a24e0 0x7f58f8071c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36986/0 (socket says 172.21.15.78:36986) 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58fe514640 1 -- 172.21.15.78:0/3685909980 learned_addr learned my addr 172.21.15.78:0/3685909980 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 client.?.objecter ms_handle_connect 0x7f58f80a24e0 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223701435 0 0) 0x7f58e8003aa0 con 0x7f58f80a24e0 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): my global_id is 28275 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58dc004ec0 con 0x7f58f80a24e0 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 client.?.objecter ms_handle_connect 0x7f58f8135b60 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1932308842 0 0) 0x7f58dc004ec0 con 0x7f58f80a24e0 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f58e8003aa0 con 0x7f58f80a24e0 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f58e80043b0 con 0x7f58f80a24e0 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): dump: 2024-09-15T20:33:42.911 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1838819490 0 0) 0x7f58f81384f0 con 0x7f58f8135b60 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:42.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient(hunting): my global_id is 22768 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f58dc004ec0 con 0x7f58f8135b60 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4115061367 0 0) 0x7f58e8003aa0 con 0x7f58f80a24e0 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 >> v1:172.21.15.78:6790/0 conn(0x7f58f813c220 legacy=0x7f58f8072a80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 >> v1:172.21.15.146:6789/0 conn(0x7f58f8135b60 legacy=0x7f58f8072370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient: _finish_hunting 0 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 monclient: found mon.a 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:42.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f8136910 con 0x7f58f80a24e0 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient: _finish_auth 0 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient: _check_auth_tickets 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 5 monclient: authenticate success, global_id 28275 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: _renew_subs 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f58f813bb20 con 0x7f58f80a24e0 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 client.28275.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: _renew_subs 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f58f81d3c20 con 0x7f58f80a24e0 2024-09-15T20:33:42.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 librados: init done 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f58e80045d0 con 0x7f58f80a24e0 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f58e8004a60 con 0x7f58f80a24e0 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58effff640 4 set_mon_vals no callback set 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f58eeffd640 10 monclient: dump: 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:42.915 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.894+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58e0005090 con 0x7f58f80a24e0 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.898+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f58e8004c00 con 0x7f58f80a24e0 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.898+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 7111+0+0 (unknown 4011571972 0 0) 0x7f58e804c020 con 0x7f58f80a24e0 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.898+0000 7f58eeffd640 10 client.28275.objecter ms_dispatch 0x7f58f8000ba0 osd_map(75..75 src has 1..75) 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.898+0000 7f58eeffd640 3 client.28275.objecter handle_osd_map got epochs [75,75] > 0 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.898+0000 7f58eeffd640 3 client.28275.objecter handle_osd_map decoding full epoch 75 2024-09-15T20:33:42.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.898+0000 7f58eeffd640 20 client.28275.objecter dump_active .. 0 homeless 2024-09-15T20:33:42.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.898+0000 7f58fdd13640 10 monclient: get_auth_request con 0x7f58dc03e080 auth_method 0 2024-09-15T20:33:42.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.898+0000 7f58eeffd640 10 client.28275.objecter ms_handle_connect 0x7f58dc03e080 2024-09-15T20:33:42.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.902+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f58f809d180 con 0x7f58f80a24e0 2024-09-15T20:33:42.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.902+0000 7f58eeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:42.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:42.902+0000 7f58eeffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:43.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.134+0000 7f5900778640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:33:43.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.134+0000 7f5900778640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:43.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.134+0000 7f5900778640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:43.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.134+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f58e0005380 con 0x7f58f80a24e0 2024-09-15T20:33:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.134+0000 7f58eeffd640 1 -- 172.21.15.78:0/3685909980 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 270395905) 0x7f58e8016c40 con 0x7f58f80a24e0 2024-09-15T20:33:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.134+0000 7f58eeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:43.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.134+0000 7f58eeffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:43.140 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 10 librados: watch_flush enter 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 10 librados: watch_flush exit 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 20 client.28275.objecter shutdown clearing up homeless session... 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 10 client.28275.objecter successfully canceled tick 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 >> v1:172.21.15.146:6812/435598642 conn(0x7f58dc03e080 legacy=0x7f58dc040540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 10 monclient: shutdown 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 >> v1:172.21.15.78:6789/0 conn(0x7f58f80a24e0 legacy=0x7f58f8071c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 shutdown_connections 2024-09-15T20:33:43.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 >> 172.21.15.78:0/3685909980 conn(0x7f58f80a63c0 msgr2=0x7f58f812e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 shutdown_connections 2024-09-15T20:33:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 1 -- 172.21.15.78:0/3685909980 wait complete. 2024-09-15T20:33:43.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:43.138+0000 7f5900778640 1 librados: shutdown 2024-09-15T20:33:46.161 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:33:46.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient: get_monmap_and_config 2024-09-15T20:33:46.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient: build_initial_monmap 2024-09-15T20:33:46.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient: monmap: 2024-09-15T20:33:46.291 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:46.291 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:46.291 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:46.290611+0000 2024-09-15T20:33:46.291 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:46.290611+0000 2024-09-15T20:33:46.292 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:46.292 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:46.292 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:46.292 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:46.292 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:46.292 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 1 Processor -- start 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 1 -- start start 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient: init 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient(hunting): picked mon.noname-b con 0x7f9564134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient(hunting): picked mon.noname-c con 0x7f956409f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient(hunting): picked mon.noname-a con 0x7f956406e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95640a0470 con 0x7f956406e960 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f956409feb0 con 0x7f956409f5a0 2024-09-15T20:33:46.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f956409e4a0 con 0x7f9564134bb0 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9569721640 10 monclient(hunting): authenticate will time out at 1287.175665s 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f9562ffd640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f9564134bb0 legacy=0x7f9564134f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f95637fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f956406e960 0x7f956413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36992/0 (socket says 172.21.15.78:36992) 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.286+0000 7f95637fe640 1 -- 172.21.15.78:0/1788718709 learned_addr learned my addr 172.21.15.78:0/1788718709 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2580830606 0 0) 0x7f955c0014a0 con 0x7f956406e960 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient(hunting): _negotiate 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient(hunting): my global_id is 28278 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9550003530 con 0x7f956406e960 2024-09-15T20:33:46.296 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 515723306 0 0) 0x7f9550003530 con 0x7f956406e960 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 >> v1:172.21.15.78:6790/0 conn(0x7f9564134bb0 legacy=0x7f9564134f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 >> v1:172.21.15.146:6789/0 conn(0x7f956409f5a0 legacy=0x7f95641399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: _finish_hunting 0 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 monclient: found mon.noname-a 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95640a0ff0 con 0x7f956406e960 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: _finish_auth 0 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: _check_auth_tickets 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f955c001bf0 con 0x7f956406e960 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 5 monclient: authenticate success, global_id 28278 2024-09-15T20:33:46.297 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: dump: 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:46.298 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f955c001e10 con 0x7f956406e960 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 1 -- 172.21.15.78:0/1788718709 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f955c004dd0 con 0x7f956406e960 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95627fc640 10 monclient: dump: 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:46.299 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 4 set_mon_vals no callback set 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: shutdown 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- 172.21.15.78:0/1788718709 >> v1:172.21.15.78:6789/0 conn(0x7f956406e960 legacy=0x7f956413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- 172.21.15.78:0/1788718709 shutdown_connections 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- 172.21.15.78:0/1788718709 >> 172.21.15.78:0/1788718709 conn(0x7f95640a63c0 msgr2=0x7f95640a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- 172.21.15.78:0/1788718709 shutdown_connections 2024-09-15T20:33:46.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- 172.21.15.78:0/1788718709 wait complete. 2024-09-15T20:33:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: build_initial_monmap 2024-09-15T20:33:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: monmap: 2024-09-15T20:33:46.304 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 librados: starting msgr at 2024-09-15T20:33:46.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 librados: starting objecter 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 Processor -- start 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- start start 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 librados: setting wanted keys 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 librados: calling monclient init 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: init 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient(hunting): picked mon.noname-c con 0x7f956406e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient(hunting): picked mon.noname-b con 0x7f956409f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient(hunting): picked mon.noname-a con 0x7f9564134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9562ffd640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f956406e960 legacy=0x7f9564133a50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95641230f0 con 0x7f9564134bb0 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f956409d180 con 0x7f956409f5a0 2024-09-15T20:33:46.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95641ccfa0 con 0x7f956406e960 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient(hunting): authenticate will time out at 1287.179103s 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95637fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9564134bb0 0x7f95641d53a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36994/0 (socket says 172.21.15.78:36994) 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f95637fe640 1 -- 172.21.15.78:0/4072910936 learned_addr learned my addr 172.21.15.78:0/4072910936 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 client.?.objecter ms_handle_connect 0x7f9564134bb0 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680205485 0 0) 0x7f95641230f0 con 0x7f9564134bb0 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient(hunting): my global_id is 28281 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9544004ec0 con 0x7f9564134bb0 2024-09-15T20:33:46.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 client.?.objecter ms_handle_connect 0x7f956409f5a0 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1982556351 0 0) 0x7f9544004ec0 con 0x7f9564134bb0 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95641230f0 con 0x7f9564134bb0 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f955c001bb0 con 0x7f9564134bb0 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient(hunting): dump: 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:46.308 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 118740863 0 0) 0x7f95641230f0 con 0x7f9564134bb0 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 >> v1:172.21.15.78:6790/0 conn(0x7f956406e960 legacy=0x7f9564133a50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 >> v1:172.21.15.146:6789/0 conn(0x7f956409f5a0 legacy=0x7f95641d1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 monclient: found mon.a 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9564125450 con 0x7f9564134bb0 2024-09-15T20:33:46.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient: _finish_auth 0 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f955bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 5 monclient: authenticate success, global_id 28281 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: _renew_subs 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.290+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9564123c50 con 0x7f9564134bb0 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 10 client.28281.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 10 monclient: _renew_subs 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9564134330 con 0x7f9564134bb0 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 1 librados: init done 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f955c001dd0 con 0x7f9564134bb0 2024-09-15T20:33:46.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f955c005230 con 0x7f9564134bb0 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9561ffb640 4 set_mon_vals no callback set 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 10 monclient: dump: 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:46.311 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9548005090 con 0x7f9564134bb0 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f955c0112a0 con 0x7f9564134bb0 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(75..75 src has 1..75) ==== 7111+0+0 (unknown 4011571972 0 0) 0x7f955c04c280 con 0x7f9564134bb0 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 10 client.28281.objecter ms_dispatch 0x7f9564000ba0 osd_map(75..75 src has 1..75) 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 3 client.28281.objecter handle_osd_map got epochs [75,75] > 0 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 3 client.28281.objecter handle_osd_map decoding full epoch 75 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 20 client.28281.objecter dump_active .. 0 homeless 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f9562ffd640 10 monclient: get_auth_request con 0x7f954403df50 auth_method 0 2024-09-15T20:33:46.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.294+0000 7f955bfff640 10 client.28281.objecter ms_handle_connect 0x7f954403df50 2024-09-15T20:33:46.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.298+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f95640a0470 con 0x7f9564134bb0 2024-09-15T20:33:46.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.298+0000 7f955bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:46.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.298+0000 7f955bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:46.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.522+0000 7f9569721640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:33:46.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.522+0000 7f9569721640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:46.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.522+0000 7f9569721640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:46.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.522+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9548005380 con 0x7f9564134bb0 2024-09-15T20:33:46.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.522+0000 7f955bfff640 1 -- 172.21.15.78:0/4072910936 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 394496792) 0x7f955c016f00 con 0x7f9564134bb0 2024-09-15T20:33:46.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.522+0000 7f955bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:46.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.522+0000 7f955bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:46.529 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":10,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:33:46.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 10 librados: watch_flush enter 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 10 librados: watch_flush exit 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 20 client.28281.objecter shutdown clearing up homeless session... 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 10 client.28281.objecter successfully canceled tick 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 >> v1:172.21.15.146:6812/435598642 conn(0x7f954403df50 legacy=0x7f9544040410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 10 monclient: shutdown 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 >> v1:172.21.15.78:6789/0 conn(0x7f9564134bb0 legacy=0x7f95641d53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 shutdown_connections 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 >> 172.21.15.78:0/4072910936 conn(0x7f95640a63c0 msgr2=0x7f95641bc3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 shutdown_connections 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 1 -- 172.21.15.78:0/4072910936 wait complete. 2024-09-15T20:33:46.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:46.526+0000 7f9569721640 1 librados: shutdown 2024-09-15T20:33:49.546 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:33:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient: get_monmap_and_config 2024-09-15T20:33:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient: build_initial_monmap 2024-09-15T20:33:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient: monmap: 2024-09-15T20:33:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:49.678 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:49.677572+0000 2024-09-15T20:33:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:49.677572+0000 2024-09-15T20:33:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:49.679 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:49.679 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:49.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 1 Processor -- start 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 1 -- start start 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient: init 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient(hunting): picked mon.noname-c con 0x7fd0ac135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ac0a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ac13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac0745e0 con 0x7fd0ac0a24e0 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac06ecf0 con 0x7fd0ac135b60 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac06efd0 con 0x7fd0ac13c220 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:49.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0b18e5640 10 monclient(hunting): authenticate will time out at 1290.562094s 2024-09-15T20:33:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0ab7fe640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd0ac13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0aa7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd0ac0a24e0 0x7fd0ac13ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36998/0 (socket says 172.21.15.78:36998) 2024-09-15T20:33:49.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.674+0000 7fd0aa7fc640 1 -- 172.21.15.78:0/1789029785 learned_addr learned my addr 172.21.15.78:0/1789029785 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:49.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:49.878+0000 7fd0ab7fe640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd0ac13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:50.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:50.282+0000 7fd0ab7fe640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd0ac13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:51.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:51.086+0000 7fd0ab7fe640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd0ac13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 10 monclient(hunting): tick 2024-09-15T20:33:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:33:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 1 monclient(hunting): continuing hunt 2024-09-15T20:33:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:33:52.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd0ac13e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:33:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.146:6789/0 conn(0x7fd0ac135b60 legacy=0x7fd0ac135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6789/0 conn(0x7fd0ac0a24e0 legacy=0x7fd0ac13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:33:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ac13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fd0ac135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ac0a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac136650 con 0x7fd0ac0a24e0 2024-09-15T20:33:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac06efd0 con 0x7fd0ac135b60 2024-09-15T20:33:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09800fea0 con 0x7fd0ac13c220 2024-09-15T20:33:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0a8ff9640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.678+0000 7fd0aaffd640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:52.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:52.882+0000 7fd0aaffd640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:53.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:53.286+0000 7fd0aaffd640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:54.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:54.090+0000 7fd0aaffd640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 10 monclient(hunting): tick 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 1 monclient(hunting): continuing hunt 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.146:6789/0 conn(0x7fd0ac135b60 legacy=0x7fd098008eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6789/0 conn(0x7fd0ac0a24e0 legacy=0x7fd09800c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7fd0980101a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7fd0980095e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:55.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7fd0ac13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:55.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0980111a0 con 0x7fd0980095e0 2024-09-15T20:33:55.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09800fea0 con 0x7fd0980101a0 2024-09-15T20:33:55.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 1 -- 172.21.15.78:0/1789029785 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd09800a0d0 con 0x7fd0ac13c220 2024-09-15T20:33:55.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0a8ff9640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:55.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.678+0000 7fd0ab7fe640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:55.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:55.882+0000 7fd0ab7fe640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:56.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:56.286+0000 7fd0ab7fe640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:57.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.090+0000 7fd0ab7fe640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.610+0000 7fd0a9ffb640 1 -- 172.21.15.78:0/1789029785 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4182495318 0 0) 0x7fd0980111a0 con 0x7fd0980095e0 2024-09-15T20:33:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.610+0000 7fd0a9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:33:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.610+0000 7fd0a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.610+0000 7fd0a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.610+0000 7fd0a9ffb640 10 monclient(hunting): my global_id is 28284 2024-09-15T20:33:57.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.610+0000 7fd0a9ffb640 1 -- 172.21.15.78:0/1789029785 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd094003530 con 0x7fd0980095e0 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 1 -- 172.21.15.78:0/1789029785 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3201478716 0 0) 0x7fd094003530 con 0x7fd0980095e0 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6790/0 conn(0x7fd0ac13c220 legacy=0x7fd098005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.146:6789/0 conn(0x7fd0980101a0 legacy=0x7fd098010580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 1 monclient: found mon.noname-a 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 1 -- 172.21.15.78:0/1789029785 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd09800a2f0 con 0x7fd0980095e0 2024-09-15T20:33:57.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0b18e5640 5 monclient: authenticate success, global_id 28284 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0b18e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 1 -- 172.21.15.78:0/1789029785 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd0a00015e0 con 0x7fd0980095e0 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 10 monclient: dump: 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:57.620 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:57.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0b18e5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:57.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 1 -- 172.21.15.78:0/1789029785 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd0a0001780 con 0x7fd0980095e0 2024-09-15T20:33:57.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:57.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0b18e5640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:57.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.614+0000 7fd0b18e5640 4 set_mon_vals no callback set 2024-09-15T20:33:57.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 10 monclient: shutdown 2024-09-15T20:33:57.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:57.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 -- 172.21.15.78:0/1789029785 >> v1:172.21.15.78:6789/0 conn(0x7fd0980095e0 legacy=0x7fd0980099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:57.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 -- 172.21.15.78:0/1789029785 shutdown_connections 2024-09-15T20:33:57.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 -- 172.21.15.78:0/1789029785 >> 172.21.15.78:0/1789029785 conn(0x7fd0ac0a63c0 msgr2=0x7fd0ac0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:57.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 -- 172.21.15.78:0/1789029785 shutdown_connections 2024-09-15T20:33:57.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 -- 172.21.15.78:0/1789029785 wait complete. 2024-09-15T20:33:57.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 10 monclient: build_initial_monmap 2024-09-15T20:33:57.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 10 monclient: monmap: 2024-09-15T20:33:57.624 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:57.624 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:57.624 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:57.624 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:57.624 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:57.625 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:57.625 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:57.625 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:57.625 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:57.625 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:57.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 librados: starting msgr at 2024-09-15T20:33:57.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 librados: starting objecter 2024-09-15T20:33:57.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 Processor -- start 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 -- start start 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 librados: setting wanted keys 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 1 librados: calling monclient init 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 10 monclient: init 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.618+0000 7fd0b18e5640 10 monclient(hunting): picked mon.noname-b con 0x7fd0980095e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0b18e5640 10 monclient(hunting): picked mon.noname-c con 0x7fd0980101a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0b18e5640 10 monclient(hunting): picked mon.noname-a con 0x7fd0ac13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0b18e5640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac13a4f0 con 0x7fd0ac13c220 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0b18e5640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac139ea0 con 0x7fd0980095e0 2024-09-15T20:33:57.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0b18e5640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd0ac11d5c0 con 0x7fd0980101a0 2024-09-15T20:33:57.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0b18e5640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:57.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0b18e5640 10 monclient(hunting): authenticate will time out at 1298.508105s 2024-09-15T20:33:57.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0aa7fc640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7fd0980101a0 legacy=0x7fd0ac1de7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:57.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0aaffd640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7fd0980095e0 0x7fd0ac139790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.78:56002/0 (socket says 172.21.15.78:56002) 2024-09-15T20:33:57.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd0aaffd640 1 -- 172.21.15.78:0/3340159140 learned_addr learned my addr 172.21.15.78:0/3340159140 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:57.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd093fff640 10 client.?.objecter ms_handle_connect 0x7fd0980095e0 2024-09-15T20:33:57.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.622+0000 7fd093fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:57.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.630+0000 7fd093fff640 10 client.?.objecter ms_handle_connect 0x7fd0ac13c220 2024-09-15T20:33:57.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.630+0000 7fd093fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:57.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.630+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1119222414 0 0) 0x7fd0ac13a4f0 con 0x7fd0ac13c220 2024-09-15T20:33:57.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.630+0000 7fd093fff640 10 monclient(hunting): _negotiate 2024-09-15T20:33:57.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.630+0000 7fd093fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:57.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.630+0000 7fd093fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:57.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.630+0000 7fd093fff640 10 monclient(hunting): my global_id is 28287 2024-09-15T20:33:57.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.630+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd098011140 con 0x7fd0ac13c220 2024-09-15T20:33:57.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2586703292 0 0) 0x7fd098011140 con 0x7fd0ac13c220 2024-09-15T20:33:57.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd0ac13a4f0 con 0x7fd0ac13c220 2024-09-15T20:33:57.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd0a4004630 con 0x7fd0ac13c220 2024-09-15T20:33:57.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:57.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:57.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient(hunting): dump: 2024-09-15T20:33:57.640 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:57.640 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:57.640 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1438670526 0 0) 0x7fd0ac13a4f0 con 0x7fd0ac13c220 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 >> v1:172.21.15.78:6790/0 conn(0x7fd0980101a0 legacy=0x7fd0ac1de7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 >> v1:172.21.15.146:6789/0 conn(0x7fd0980095e0 legacy=0x7fd0ac139790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient: _finish_hunting 0 2024-09-15T20:33:57.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 monclient: found mon.a 2024-09-15T20:33:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0ac11cd20 con 0x7fd0ac13c220 2024-09-15T20:33:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient: _finish_auth 0 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient: _check_auth_tickets 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 5 monclient: authenticate success, global_id 28287 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 10 monclient: _renew_subs 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd0ac134260 con 0x7fd0ac13c220 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 10 client.28287.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 10 monclient: _renew_subs 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:57.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd0ac13b220 con 0x7fd0ac13c220 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0b18e5640 1 librados: init done 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd0a4004b40 con 0x7fd0ac13c220 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd093fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.634+0000 7fd0a97fa640 4 set_mon_vals no callback set 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd0a4004f10 con 0x7fd0ac13c220 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 10 monclient: dump: 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:57.644 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:57.645 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:57.645 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:57.645 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:57.645 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:57.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd0a4005170 con 0x7fd0ac13c220 2024-09-15T20:33:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 6603+0+0 (unknown 2084286170 0 0) 0x7fd0a404bf40 con 0x7fd0ac13c220 2024-09-15T20:33:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 10 client.28287.objecter ms_dispatch 0x7fd0ac000ba0 osd_map(76..76 src has 1..76) 2024-09-15T20:33:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 3 client.28287.objecter handle_osd_map got epochs [76,76] > 0 2024-09-15T20:33:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 3 client.28287.objecter handle_osd_map decoding full epoch 76 2024-09-15T20:33:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd093fff640 20 client.28287.objecter dump_active .. 0 homeless 2024-09-15T20:33:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.638+0000 7fd0aaffd640 10 monclient: get_auth_request con 0x7fd09803f150 auth_method 0 2024-09-15T20:33:57.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.642+0000 7fd093fff640 10 client.28287.objecter ms_handle_connect 0x7fd09803f150 2024-09-15T20:33:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.646+0000 7fd0b18e5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.646+0000 7fd0b18e5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.646+0000 7fd0b18e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:57.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.646+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd08c005090 con 0x7fd0ac13c220 2024-09-15T20:33:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.658+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd0a4016d80 con 0x7fd0ac13c220 2024-09-15T20:33:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.658+0000 7fd093fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:57.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.658+0000 7fd093fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: Running main() from gmock_main.cc 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [==========] Running 13 tests from 4 test suites. 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] Global test environment set-up. 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapList 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapList (1967 ms) 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapRemove 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapRemove (2045 ms) 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.Rollback 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshots.Rollback (1993 ms) 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshots.SnapGetName 2024-09-15T20:33:57.675 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshots.SnapGetName (2023 ms) 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshots (8030 ms total) 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Snap 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Snap (3736 ms) 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.Rollback 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.Rollback (4019 ms) 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManaged.FutureSnapRollback 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManaged.FutureSnapRollback (4985 ms) 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] 3 tests from LibRadosSnapshotsSelfManaged (12740 ms total) 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapList 2024-09-15T20:33:57.676 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapList (2959 ms) 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapRemove 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapRemove (1975 ms) 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.Rollback 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.Rollback (2040 ms) 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsEC.SnapGetName 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsEC.SnapGetName (2073 ms) 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] 4 tests from LibRadosSnapshotsEC (9047 ms total) 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Snap 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Snap (4053 ms) 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ RUN ] LibRadosSnapshotsSelfManagedEC.Rollback 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ OK ] LibRadosSnapshotsSelfManagedEC.Rollback (3984 ms) 2024-09-15T20:33:57.677 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] 2 tests from LibRadosSnapshotsSelfManagedEC (8037 ms total) 2024-09-15T20:33:57.678 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: 2024-09-15T20:33:57.678 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [----------] Global test environment tear-down 2024-09-15T20:33:57.678 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [==========] 13 tests from 4 test suites ran. (68998 ms total) 2024-09-15T20:33:57.678 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots: [ PASSED ] 13 tests. 2024-09-15T20:33:57.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.894+0000 7fd0b18e5640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:33:57.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.894+0000 7fd0b18e5640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:57.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.894+0000 7fd0b18e5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:57.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.894+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd08c005380 con 0x7fd0ac13c220 2024-09-15T20:33:57.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.894+0000 7fd093fff640 1 -- 172.21.15.78:0/3340159140 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 2431665871) 0x7fd0a4016680 con 0x7fd0ac13c220 2024-09-15T20:33:57.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.894+0000 7fd093fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:33:57.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.894+0000 7fd093fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:57.900 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":14,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 10 librados: watch_flush enter 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 10 librados: watch_flush exit 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 20 client.28287.objecter shutdown clearing up homeless session... 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 10 client.28287.objecter successfully canceled tick 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 >> v1:172.21.15.146:6812/435598642 conn(0x7fd09803f150 legacy=0x7fd098041610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 10 monclient: shutdown 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 >> v1:172.21.15.78:6789/0 conn(0x7fd0ac13c220 legacy=0x7fd0ac1e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:57.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 shutdown_connections 2024-09-15T20:33:57.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 >> 172.21.15.78:0/3340159140 conn(0x7fd0ac0a63c0 msgr2=0x7fd0ac134f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:57.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 shutdown_connections 2024-09-15T20:33:57.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 1 -- 172.21.15.78:0/3340159140 wait complete. 2024-09-15T20:33:57.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:57.898+0000 7fd0b18e5640 1 librados: shutdown 2024-09-15T20:33:57.919 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:33:57.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T20:33:57.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:33:58.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient: get_monmap_and_config 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient: build_initial_monmap 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient: monmap: 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:58.037210+0000 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:58.037210+0000 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:58.038 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:58.039 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 Processor -- start 2024-09-15T20:33:58.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- start start 2024-09-15T20:33:58.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient: init 2024-09-15T20:33:58.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:58.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:33:58.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient(hunting): picked mon.noname-c con 0x7f83240a1a90 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient(hunting): picked mon.noname-b con 0x7f83240a3e00 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient(hunting): picked mon.noname-a con 0x7f832413dcd0 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f832406e3d0 con 0x7f832413dcd0 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f832406e6b0 con 0x7f83240a1a90 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8324143800 con 0x7f83240a3e00 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient(hunting): authenticate will time out at 1298.921743s 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f83237fe640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f83240a3e00 legacy=0x7f83240a1e70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8328934640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f832413dcd0 0x7f83241400c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49204/0 (socket says 172.21.15.78:49204) 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8328934640 1 -- 172.21.15.78:0/4132016309 learned_addr learned my addr 172.21.15.78:0/4132016309 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2649728179 0 0) 0x7f832406e3d0 con 0x7f832413dcd0 2024-09-15T20:33:58.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient(hunting): my global_id is 28299 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f830c003530 con 0x7f832413dcd0 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2367823658 0 0) 0x7f830c003530 con 0x7f832413dcd0 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 >> v1:172.21.15.78:6790/0 conn(0x7f83240a3e00 legacy=0x7f83240a1e70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 >> v1:172.21.15.146:6789/0 conn(0x7f83240a1a90 legacy=0x7f8324134960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 monclient: found mon.noname-a 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83241449e0 con 0x7f832413dcd0 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: _finish_auth 0 2024-09-15T20:33:58.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f831c001a20 con 0x7f832413dcd0 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: dump: 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.044 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 5 monclient: authenticate success, global_id 28299 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f831c001bc0 con 0x7f832413dcd0 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 1 -- 172.21.15.78:0/4132016309 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f831c001fd0 con 0x7f832413dcd0 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f8322ffd640 10 monclient: dump: 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.045 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 4 set_mon_vals no callback set 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 10 monclient: shutdown 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- 172.21.15.78:0/4132016309 >> v1:172.21.15.78:6789/0 conn(0x7f832413dcd0 legacy=0x7f83241400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- 172.21.15.78:0/4132016309 shutdown_connections 2024-09-15T20:33:58.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- 172.21.15.78:0/4132016309 >> 172.21.15.78:0/4132016309 conn(0x7f83240a4c90 msgr2=0x7f83240a50a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- 172.21.15.78:0/4132016309 shutdown_connections 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.034+0000 7f832a397640 1 -- 172.21.15.78:0/4132016309 wait complete. 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: build_initial_monmap 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: monmap: 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:58.047 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 librados: starting msgr at 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 librados: starting objecter 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 Processor -- start 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 -- start start 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 librados: setting wanted keys 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 librados: calling monclient init 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: init 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient(hunting): picked mon.noname-b con 0x7f83240a1a90 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient(hunting): picked mon.noname-a con 0x7f83240a3e00 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient(hunting): picked mon.noname-c con 0x7f832413dcd0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:58.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8324140770 con 0x7f83240a3e00 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f832413eed0 con 0x7f83240a1a90 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8324140a50 con 0x7f832413dcd0 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8328934640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f832413dcd0 legacy=0x7f83241d4940 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient(hunting): authenticate will time out at 1298.925316s 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f83237fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f83240a3e00 0x7f832413e7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49220/0 (socket says 172.21.15.78:49220) 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f83237fe640 1 -- 172.21.15.78:0/2454567674 learned_addr learned my addr 172.21.15.78:0/2454567674 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 client.?.objecter ms_handle_connect 0x7f83240a3e00 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1577466751 0 0) 0x7f8324140770 con 0x7f83240a3e00 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): my global_id is 28302 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8310004ec0 con 0x7f83240a3e00 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 client.?.objecter ms_handle_connect 0x7f83240a1a90 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1097302310 0 0) 0x7f8310004ec0 con 0x7f83240a3e00 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8324140770 con 0x7f83240a3e00 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f83140041e0 con 0x7f83240a3e00 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): dump: 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.055 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1400462268 0 0) 0x7f8324140770 con 0x7f83240a3e00 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 >> v1:172.21.15.78:6790/0 conn(0x7f832413dcd0 legacy=0x7f83241d4940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 >> v1:172.21.15.146:6789/0 conn(0x7f83240a1a90 legacy=0x7f832413e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 monclient: found mon.a 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:58.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f832413da40 con 0x7f83240a3e00 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: _finish_auth 0 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 5 monclient: authenticate success, global_id 28302 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: _renew_subs 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8324140490 con 0x7f83240a3e00 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 client.28302.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: _renew_subs 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f83241310d0 con 0x7f83240a3e00 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 librados: init done 2024-09-15T20:33:58.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8314004400 con 0x7f83240a3e00 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8314004810 con 0x7f83240a3e00 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f83227fc640 4 set_mon_vals no callback set 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: dump: 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.058 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f83040065d0 con 0x7f8304002870 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 client.28302.objecter ms_handle_connect 0x7f8304002870 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 client.28302.objecter resend_mon_ops 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 490143385 0 0) 0x7f83040065d0 con 0x7f8304002870 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient(hunting): my global_id is 28305 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8324140770 con 0x7f8304002870 2024-09-15T20:33:58.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 268771841 0 0) 0x7f8324140770 con 0x7f8304002870 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f83040065d0 con 0x7f8304002870 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f8314004a30 con 0x7f83240a3e00 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f83180017d0 con 0x7f8304002870 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 monclient: dump: 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.060 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 6603+0+0 (unknown 2084286170 0 0) 0x7f831404bce0 con 0x7f83240a3e00 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 10 client.28302.objecter ms_dispatch 0x7f8324000ba0 osd_map(76..76 src has 1..76) 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 3 client.28302.objecter handle_osd_map got epochs [76,76] > 0 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 3 client.28302.objecter handle_osd_map decoding full epoch 76 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 20 client.28302.objecter dump_active .. 0 homeless 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2616703081 0 0) 0x7f83040065d0 con 0x7f8304002870 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.038+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f830400a170 con 0x7f8304002870 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.042+0000 7f8328934640 10 monclient: get_auth_request con 0x7f831003e440 auth_method 0 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.042+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8318001b60 con 0x7f8304002870 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.042+0000 7f8320ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.042+0000 7f8320ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.042+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 >> v1:172.21.15.78:6789/0 conn(0x7f8304002870 legacy=0x7f8304004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.042+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 >> v1:172.21.15.78:6789/0 conn(0x7f8304002870 legacy=0x7f8304004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.042+0000 7f8320ff9640 10 client.28302.objecter ms_handle_connect 0x7f831003e440 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8304009e90 con 0x7f8304009150 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 client.28302.objecter ms_handle_connect 0x7f8304009150 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 client.28302.objecter resend_mon_ops 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 576222945 0 0) 0x7f8304009e90 con 0x7f8304009150 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient(hunting): my global_id is 28308 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83040065d0 con 0x7f8304009150 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1205273081 0 0) 0x7f83040065d0 con 0x7f8304009150 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8304009e90 con 0x7f8304009150 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f83180037c0 con 0x7f8304009150 2024-09-15T20:33:58.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient: dump: 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 374406941 0 0) 0x7f8304009e90 con 0x7f8304009150 2024-09-15T20:33:58.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f830400c420 con 0x7f8304009150 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4129 (unknown 0 0 3822938760) 0x7f8318003750 con 0x7f8304009150 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 >> v1:172.21.15.78:6789/0 conn(0x7f8304009150 legacy=0x7f83040095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8320ff9640 1 -- 172.21.15.78:0/2454567674 >> v1:172.21.15.78:6789/0 conn(0x7f8304009150 legacy=0x7f83040095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 14, 2024-09-15T20:33:58.064 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 150597, 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:58.065 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:58.066 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:33:58.067 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:58.068 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.069 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:58.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:58.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.073 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:58.074 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:58.075 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:33:58.076 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.076 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.076 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.076 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:33:58.076 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:33:58.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 10 librados: watch_flush enter 2024-09-15T20:33:58.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 10 librados: watch_flush exit 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 20 client.28302.objecter shutdown clearing up homeless session... 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 10 client.28302.objecter successfully canceled tick 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 >> v1:172.21.15.146:6812/435598642 conn(0x7f831003e440 legacy=0x7f8310040900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 10 monclient: shutdown 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 >> v1:172.21.15.78:6789/0 conn(0x7f83240a3e00 legacy=0x7f832413e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8328934640 1 -- 172.21.15.78:0/2454567674 reap_dead start 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f8328934640 1 -- 172.21.15.78:0/2454567674 reap_dead start 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 shutdown_connections 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 >> 172.21.15.78:0/2454567674 conn(0x7f83240a4c90 msgr2=0x7f83240a6960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 shutdown_connections 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 1 -- 172.21.15.78:0/2454567674 wait complete. 2024-09-15T20:33:58.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.046+0000 7f832a397640 1 librados: shutdown 2024-09-15T20:33:58.079 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:33:58.079 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T20:33:58.079 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:33:58.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient: get_monmap_and_config 2024-09-15T20:33:58.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient: build_initial_monmap 2024-09-15T20:33:58.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient: monmap: 2024-09-15T20:33:58.190 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:58.190 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.190 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:33:58.189613+0000 2024-09-15T20:33:58.190 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:33:58.189613+0000 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 1 Processor -- start 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 1 -- start start 2024-09-15T20:33:58.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient: init 2024-09-15T20:33:58.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:58.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient(hunting): picked mon.noname-a con 0x7f2be409ca50 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient(hunting): picked mon.noname-b con 0x7f2be409d540 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient(hunting): picked mon.noname-c con 0x7f2be40a56f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be40a6c50 con 0x7f2be409ca50 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be40a4d50 con 0x7f2be40a56f0 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be40a8ca0 con 0x7f2be409d540 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient(hunting): authenticate will time out at 1299.074977s 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be1d9b640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f2be409d540 legacy=0x7f2be413cf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be259c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2be409ca50 0x7f2be409ce30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49252/0 (socket says 172.21.15.78:49252) 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be259c640 1 -- 172.21.15.78:0/413060448 learned_addr learned my addr 172.21.15.78:0/413060448 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:58.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 387358496 0 0) 0x7f2be40a6c50 con 0x7f2be409ca50 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient(hunting): my global_id is 28311 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bcc003530 con 0x7f2be409ca50 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 400218072 0 0) 0x7f2bcc003530 con 0x7f2be409ca50 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 >> v1:172.21.15.78:6790/0 conn(0x7f2be409d540 legacy=0x7f2be413cf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 >> v1:172.21.15.146:6789/0 conn(0x7f2be40a56f0 legacy=0x7f2be4140690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: _finish_hunting 0 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 monclient: found mon.noname-a 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:33:58.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be40a5b40 con 0x7f2be409ca50 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: _finish_auth 0 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: _check_auth_tickets 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2bd40016a0 con 0x7f2be409ca50 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 5 monclient: authenticate success, global_id 28311 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: dump: 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.197 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2bd40018c0 con 0x7f2be409ca50 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 1 -- 172.21.15.78:0/413060448 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2bd4001d50 con 0x7f2be409ca50 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:58.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be159a640 10 monclient: dump: 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 10 monclient: get_monmap_and_config success 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.186+0000 7f2be8818640 4 set_mon_vals no callback set 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: shutdown 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:58.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- 172.21.15.78:0/413060448 >> v1:172.21.15.78:6789/0 conn(0x7f2be409ca50 legacy=0x7f2be409ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- 172.21.15.78:0/413060448 shutdown_connections 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- 172.21.15.78:0/413060448 >> 172.21.15.78:0/413060448 conn(0x7f2be40aa520 msgr2=0x7f2be40aa930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- 172.21.15.78:0/413060448 shutdown_connections 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- 172.21.15.78:0/413060448 wait complete. 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: build_initial_monmap 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: monmap: 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:33:58.200 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 librados: starting msgr at 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 librados: starting objecter 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 Processor -- start 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- start start 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 librados: setting wanted keys 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 librados: calling monclient init 2024-09-15T20:33:58.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: init 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: _reopen_session rank -1 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient(hunting): picked mon.noname-b con 0x7f2be409ca50 addr v1:172.21.15.146:6789/0 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient(hunting): picked mon.noname-a con 0x7f2be409d540 addr v1:172.21.15.78:6789/0 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient(hunting): picked mon.noname-c con 0x7f2be40a56f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be4140280 con 0x7f2be409d540 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be413df20 con 0x7f2be409ca50 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2be413f9e0 con 0x7f2be40a56f0 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient(hunting): _renew_subs 2024-09-15T20:33:58.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient(hunting): authenticate will time out at 1299.078036s 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be2d9d640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f2be40a56f0 legacy=0x7f2be41d4f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be1d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2be409d540 0x7f2be41d17d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49264/0 (socket says 172.21.15.78:49264) 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be1d9b640 1 -- 172.21.15.78:0/2306817455 learned_addr learned my addr 172.21.15.78:0/2306817455 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f2be409d540 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3793324135 0 0) 0x7f2be4140280 con 0x7f2be409d540 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): my global_id is 28314 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bc4004ec0 con 0x7f2be409d540 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 client.?.objecter ms_handle_connect 0x7f2be409ca50 2024-09-15T20:33:58.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 979718425 0 0) 0x7f2bc4004ec0 con 0x7f2be409d540 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2be4140280 con 0x7f2be409d540 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2bd80041e0 con 0x7f2be409d540 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): dump: 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.208 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 628301692 0 0) 0x7f2be413df20 con 0x7f2be409ca50 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient(hunting): my global_id is 22792 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bc4004ec0 con 0x7f2be409ca50 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 811979435 0 0) 0x7f2be4140280 con 0x7f2be409d540 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 >> v1:172.21.15.78:6790/0 conn(0x7f2be40a56f0 legacy=0x7f2be41d4f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 >> v1:172.21.15.146:6789/0 conn(0x7f2be409ca50 legacy=0x7f2be4137fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:33:58.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 monclient: found mon.a 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2be413ea80 con 0x7f2be409d540 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient: _finish_auth 0 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 5 monclient: authenticate success, global_id 28314 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: _renew_subs 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2be41386e0 con 0x7f2be409d540 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 client.28314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: _renew_subs 2024-09-15T20:33:58.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2be4140bb0 con 0x7f2be409d540 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 librados: init done 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2bd8004400 con 0x7f2be409d540 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2bd8004890 con 0x7f2be409d540 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be0d99640 4 set_mon_vals no callback set 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 10 monclient: dump: 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.211 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2bc80065d0 con 0x7f2bc8002870 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.190+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f2bd8004ab0 con 0x7f2be409d540 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 client.28314.objecter ms_handle_connect 0x7f2bc8002870 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 client.28314.objecter resend_mon_ops 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(76..76 src has 1..76) ==== 6603+0+0 (unknown 2084286170 0 0) 0x7f2bd804bde0 con 0x7f2be409d540 2024-09-15T20:33:58.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 client.28314.objecter ms_dispatch 0x7f2be4000ba0 osd_map(76..76 src has 1..76) 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 3 client.28314.objecter handle_osd_map got epochs [76,76] > 0 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 3 client.28314.objecter handle_osd_map decoding full epoch 76 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 20 client.28314.objecter dump_active .. 0 homeless 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 645526215 0 0) 0x7f2bc80065d0 con 0x7f2bc8002870 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient(hunting): my global_id is 22795 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2be4140280 con 0x7f2bc8002870 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2be2d9d640 10 monclient: get_auth_request con 0x7f2bc403e030 auth_method 0 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4131918587 0 0) 0x7f2be4140280 con 0x7f2bc8002870 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bc80065d0 con 0x7f2bc8002870 2024-09-15T20:33:58.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2bd4001660 con 0x7f2bc8002870 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient: dump: 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 client.28314.objecter ms_handle_connect 0x7f2bc403e030 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2117796542 0 0) 0x7f2bc80065d0 con 0x7f2bc8002870 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2bc800a170 con 0x7f2bc8002870 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2bd4001770 con 0x7f2bc8002870 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 >> v1:172.21.15.146:6789/0 conn(0x7f2bc8002870 legacy=0x7f2bc8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.194+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 >> v1:172.21.15.146:6789/0 conn(0x7f2bc8002870 legacy=0x7f2bc8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.198+0000 7f2be8818640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.198+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2bc8009e80 con 0x7f2bc8009140 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 client.28314.objecter ms_handle_connect 0x7f2bc8009140 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 client.28314.objecter resend_mon_ops 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1217968192 0 0) 0x7f2bc8009e80 con 0x7f2bc8009140 2024-09-15T20:33:58.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:33:58.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:33:58.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:33:58.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient(hunting): my global_id is 22798 2024-09-15T20:33:58.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2bc80065d0 con 0x7f2bc8009140 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1625721750 0 0) 0x7f2bc80065d0 con 0x7f2bc8009140 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2bc8009e80 con 0x7f2bc8009140 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2bd4004b70 con 0x7f2bc8009140 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient: dump: 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.217 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1064743235 0 0) 0x7f2bc8009e80 con 0x7f2bc8009140 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2bc800c420 con 0x7f2bc8009140 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4295 (unknown 0 0 3792444228) 0x7f2bd4004b00 con 0x7f2bc8009140 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 >> v1:172.21.15.146:6789/0 conn(0x7f2bc8009140 legacy=0x7f2bc80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2bd2ffd640 1 -- 172.21.15.78:0/2306817455 >> v1:172.21.15.146:6789/0 conn(0x7f2bc8009140 legacy=0x7f2bc80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 14, 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 150710, 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:33:58.219 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:58.220 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:58.221 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:33:58.222 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:33:58.223 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.224 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.225 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:58.226 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:33:58.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.228 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:33:58.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 11 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:33:58.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:33:58.231 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:33:58.231 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:33:58.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 10 librados: watch_flush enter 2024-09-15T20:33:58.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 10 librados: watch_flush exit 2024-09-15T20:33:58.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 20 client.28314.objecter shutdown clearing up homeless session... 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 10 client.28314.objecter successfully canceled tick 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 >> v1:172.21.15.146:6812/435598642 conn(0x7f2bc403e030 legacy=0x7f2bc40404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 10 monclient: shutdown 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 >> v1:172.21.15.78:6789/0 conn(0x7f2be409d540 legacy=0x7f2be41d17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be2d9d640 1 -- 172.21.15.78:0/2306817455 reap_dead start 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be2d9d640 1 -- 172.21.15.78:0/2306817455 reap_dead start 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 shutdown_connections 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 >> 172.21.15.78:0/2306817455 conn(0x7f2be40aa520 msgr2=0x7f2be40a75a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 shutdown_connections 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 1 -- 172.21.15.78:0/2306817455 wait complete. 2024-09-15T20:33:58.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:33:58.202+0000 7f2be8818640 1 librados: shutdown 2024-09-15T20:33:58.234 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:33:58.234 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:34:03.535 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [==========] Running 31 tests from 7 test suites. 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] Global test environment set-up. 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscVersion.VersionPP 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscVersion.VersionPP (0 ms) 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscVersion (0 ms total) 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 22 tests from LibRadosMiscPP 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: seed 28280 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WaitOSDMapPP 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WaitOSDMapPP (29 ms) 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNamePP 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNamePP (198 ms) 2024-09-15T20:34:03.536 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongLocatorPP 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongLocatorPP (11 ms) 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongNSpacePP 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongNSpacePP (12 ms) 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.LongAttrNamePP 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.LongAttrNamePP (18 ms) 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.ExecPP 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.ExecPP (2 ms) 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BadFlagsPP 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BadFlagsPP (2 ms) 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate1PP 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate1PP (6 ms) 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Operate2PP 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Operate2PP (2 ms) 2024-09-15T20:34:03.537 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigObjectPP 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigObjectPP (18 ms) 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AioOperatePP 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AioOperatePP (2 ms) 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertExistsPP 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertExistsPP (5 ms) 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.AssertVersionPP 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.AssertVersionPP (26 ms) 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.BigAttrPP 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: osd_max_attr_size = 0 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: osd_max_attr_size == 0; skipping test 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.BigAttrPP (3988 ms) 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyPP 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyPP (278 ms) 2024-09-15T20:34:03.538 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CopyScrubPP 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: waiting for initial deep scrubs... 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: done waiting, doing copies 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: waiting for final deep scrubs... 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: done waiting 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CopyScrubPP (60681 ms) 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.WriteSamePP 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.WriteSamePP (14 ms) 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.CmpExtPP 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.CmpExtPP (7 ms) 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Applications 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Applications (7440 ms) 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatOSD 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatOSD (0 ms) 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.MinCompatClient 2024-09-15T20:34:03.539 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.MinCompatClient (0 ms) 2024-09-15T20:34:03.540 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscPP.Conf 2024-09-15T20:34:03.540 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscPP.Conf (0 ms) 2024-09-15T20:34:12.527 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 22 tests from LibRadosMis api_snapshots_pp: Running main() from gmock_main.cc 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [==========] Running 21 tests from 5 test suites. 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] Global test environment set-up. 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: seed 28413 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapListPP 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapListPP (1996 ms) 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapRemovePP 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapRemovePP (2046 ms) 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.RollbackPP 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.RollbackPP (1991 ms) 2024-09-15T20:34:12.528 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapGetNamePP 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapGetNamePP (2022 ms) 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsPP.SnapCreateRemovePP 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsPP.SnapCreateRemovePP (4018 ms) 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 5 tests from LibRadosSnapshotsPP (12073 ms total) 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapPP 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapPP (4216 ms) 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.RollbackPP 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.RollbackPP (3768 ms) 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.SnapOverlapPP (6026 ms) 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.Bug11677 2024-09-15T20:34:12.529 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.Bug11677 (3989 ms) 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.OrderSnap 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.OrderSnap (2132 ms) 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: ./src/test/librados/snapshots_cxx.cc:460: Skipped 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback (0 ms) 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: deleting snap 14 in pool LibRadosSnapshotsSelfManagedPP_smithi146-28413-7 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: waiting for snaps to purge 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedPP.ReusePurgedSnap (17915 ms) 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 7 tests from LibRadosSnapshotsSelfManagedPP (38046 ms total) 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: 2024-09-15T20:34:12.530 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.NotConnected (34 ms) 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosPoolIsInSelfmanagedSnapsMode.FreshInstance (4907 ms) 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 2 tests from LibRadosPoolIsInSelfmanagedSnapsMode (4941 ms total) 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapListPP 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapListPP (2989 ms) 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.SnapRemovePP 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapRemovePP (2028 ms) 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsECPP.RollbackPP 2024-09-15T20:34:12.531 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.RollbackPP (1975 ms) 2024-09-15T20:34:17.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_ snapshots: Running main() from gmock_main.cc 2024-09-15T20:34:17.559 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [==========] Running 11 tests from 2 test suites. 2024-09-15T20:34:17.559 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [----------] Global test environment set-up. 2024-09-15T20:34:17.559 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots 2024-09-15T20:34:17.559 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapList 2024-09-15T20:34:17.559 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapList (4864 ms) 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapRemove 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapRemove (5017 ms) 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSnapshots.Rollback 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSnapshots.Rollback (3679 ms) 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapGetName 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapGetName (5005 ms) 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSnapshots.SnapCreateRemove 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSnapshots.SnapCreateRemove (7033 ms) 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [----------] 5 tests from NeoRadosSnapshots (25598 ms total) 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Snap 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Snap (5026 ms) 2024-09-15T20:34:17.560 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Rollback 2024-09-15T20:34:17.561 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Rollback (5971 ms) 2024-09-15T20:34:17.561 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.SnapOverlap 2024-09-15T20:34:17.561 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.SnapOverlap (7963 ms) 2024-09-15T20:34:17.564 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.Bug11677 2024-09-15T20:34:17.564 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.Bug11677 (5998 ms) 2024-09-15T20:34:17.564 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.OrderSnap 2024-09-15T20:34:17.564 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.OrderSnap (18094 ms) 2024-09-15T20:34:17.564 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ RUN ] NeoRadosSelfManagedSnaps.ReusePurgedSnap 2024-09-15T20:34:17.564 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: Deleting snap 3 in pool ReusePurgedSnapsmithi146-29133-11. 2024-09-15T20:34:17.564 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: Waiting for snaps to purge. 2024-09-15T20:34:17.564 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ OK ] NeoRadosSelfManagedSnaps.ReusePurgedSnap (19924 ms) 2024-09-15T20:34:17.565 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [----------] 6 tests from NeoRadosSelfManagedSnaps (62976 ms total) 2024-09-15T20:34:17.565 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: 2024-09-15T20:34:17.565 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [----------] Global test environment tear-down 2024-09-15T20:34:17.565 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [==========] 11 tests from 2 test suites ran. (88574 ms total) 2024-09-15T20:34:17.565 INFO:tasks.workunit.client.0.smithi146.stdout: snapshots: [ PASSED ] 11 tests. 2024-09-15T20:34:27.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: Running main() from gmock_main.cc 2024-09-15T20:34:27.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [==========] Running 42 tests from 2 test suites. 2024-09-15T20:34:27.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [----------] Global test environment set-up. 2024-09-15T20:34:27.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [----------] 26 tests from LibRadosAio 2024-09-15T20:34:27.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.TooBig 2024-09-15T20:34:27.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.TooBig (2234 ms) 2024-09-15T20:34:27.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.SimpleWrite 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.SimpleWrite (3027 ms) 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.WaitForSafe 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.WaitForSafe (3012 ms) 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip (3005 ms) 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip2 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip2 (2646 ms) 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.RoundTrip3 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.RoundTrip3 (3014 ms) 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripAppend 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.RoundTripAppend (3058 ms) 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.RemoveTest 2024-09-15T20:34:27.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.RemoveTest (3002 ms) 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.XattrsRoundTrip 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.XattrsRoundTrip (2957 ms) 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.RmXattr 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.RmXattr (2966 ms) 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.XattrIter 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.XattrIter (3053 ms) 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.IsComplete 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.IsComplete (2952 ms) 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.IsSafe 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.IsSafe (3118 ms) 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.ReturnValue 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.ReturnValue (2901 ms) 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.Flush 2024-09-15T20:34:27.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.Flush (2990 ms) 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.FlushAsync 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.FlushAsync (3044 ms) 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteFull 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteFull (2967 ms) 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.RoundTripWriteSame 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.RoundTripWriteSame (3019 ms) 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStat 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.SimpleStat (18072 ms) 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.OperateMtime 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.OperateMtime (2911 ms) 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.Operate2Mtime 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.Operate2Mtime (3011 ms) 2024-09-15T20:34:27.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.SimpleStatNS 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.SimpleStatNS (2991 ms) 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.StatRemove 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.StatRemove (3022 ms) 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.ExecuteClass 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.ExecuteClass (2993 ms) 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.MultiWrite 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.MultiWrite (3006 ms) 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAio.AioUnlock 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAio.AioUnlock (3011 ms) 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [----------] 26 tests from LibRadosAio (91983 ms total) 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: 2024-09-15T20:34:27.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [----------] 16 tests from LibRadosAioEC 2024-09-15T20:34:27.596 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleWrite 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout:cPP (72739 ms total) 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosTwoPoolsECPP.CopyFrom 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosTwoPoolsECPP.CopyFrom (186 ms) 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 1 test from LibRadosTwoPoolsECPP (186 ms total) 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)0, Checksummer::xxhash32, ceph_le > 2024-09-15T20:34:29.591 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Subset 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Subset (65 ms) 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/0.Chunked 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosChecksum/0.Chunked (3 ms) 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/0 (68 ms total) 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)1, Checksummer::xxhash64, ceph_le > 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Subset 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Subset (48 ms) 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/1.Chunked 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosChecksum/1.Chunked (7 ms) 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/1 (55 ms total) 2024-09-15T20:34:29.592 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2, where TypeParam = LibRadosChecksumParams<(rados_checksum_type_t)2, Checksummer::crc32c, ceph_le > 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Subset 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Subset (56 ms) 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosChecksum/2.Chunked 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosChecksum/2.Chunked (6 ms) 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 2 tests from LibRadosChecksum/2 (62 ms total) 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ RUN ] LibRadosMiscECPP.CompareExtentRange 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ OK ] LibRadosMiscECPP.CompareExtentRange (1046 ms) 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] 1 test from LibRadosMiscECPP (1046 ms total) 2024-09-15T20:34:29.593 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: 2024-09-15T20:34:29.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [----------] Global test environment tear-down 2024-09-15T20:34:29.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [==========] 31 tests from 7 test suites ran. (100911 ms total) 2024-09-15T20:34:29.594 INFO:tasks.workunit.client.0.smithi146.stdout: api_misc_pp: [ PASSED ] 31 tests. 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout:pp: [ RUN ] LibRadosSnapshotsECPP.SnapGetNamePP 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsECPP.SnapGetNamePP (2016 ms) 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 4 tests from LibRadosSnapshotsECPP (9008 ms total) 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.SnapPP 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.SnapPP (4006 ms) 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.RollbackPP 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.RollbackPP (4005 ms) 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ RUN ] LibRadosSnapshotsSelfManagedECPP.Bug11677 2024-09-15T20:34:35.595 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ OK ] LibRadosSnapshotsSelfManagedECPP.Bug11677 (3963 ms) 2024-09-15T20:34:35.596 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] 3 tests from LibRadosSnapshotsSelfManagedECPP (11974 ms total) 2024-09-15T20:34:35.596 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: 2024-09-15T20:34:35.596 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [----------] Global test environment tear-down 2024-09-15T20:34:35.596 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [==========] 21 tests from 5 test suites ran. (106908 ms total) 2024-09-15T20:34:35.596 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ PASSED ] 20 tests. 2024-09-15T20:34:35.596 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ SKIPPED ] 1 test, listed below: 2024-09-15T20:34:35.596 INFO:tasks.workunit.client.0.smithi146.stdout: api_snapshots_pp: [ SKIPPED ] LibRadosSnapshotsSelfManagedPP.WriteRollback 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: Running main() from gmock_main.cc 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [==========] Running 57 tests from 4 test suites. 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] Global test environment set-up. 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.TooBigPP 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.TooBigPP (2226 ms) 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolQuotaPP 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolQuotaPP (18766 ms) 2024-09-15T20:34:41.620 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleWritePP 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleWritePP (5967 ms) 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.WaitForSafePP 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.WaitForSafePP (2975 ms) 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP (3022 ms) 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP2 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP2 (3009 ms) 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripPP3 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripPP3 (2996 ms) 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripSparseReadPP 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripSparseReadPP (3007 ms) 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsCompletePP 2024-09-15T20:34:41.621 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.IsCompletePP (2985 ms) 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.IsSafePP 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.IsSafePP (3030 ms) 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.ReturnValuePP 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.ReturnValuePP (2965 ms) 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushPP 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushPP (2986 ms) 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.FlushAsyncPP 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.FlushAsyncPP (17102 ms) 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP (2910 ms) 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteFullPP2 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteFullPP2 (3010 ms) 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP (2991 ms) 2024-09-15T20:34:41.622 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripWriteSamePP2 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripWriteSamePP2 (3022 ms) 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPPNS 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPPNS (2992 ms) 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.SimpleStatPP 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.SimpleStatPP (2996 ms) 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime (3006 ms) 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.OperateMtime2 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.OperateMtime2 (3002 ms) 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.StatRemovePP 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.StatRemovePP (3015 ms) 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.ExecuteClassPP 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.ExecuteClassPP (3018 ms) 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.OmapPP 2024-09-15T20:34:41.623 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.OmapPP (2956 ms) 2024-09-15T20:34:41.624 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiWritePP 2024-09-15T20:34:41.624 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiWritePP (3020 ms) 2024-09-15T20:34:41.624 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.AioUnlockPP 2024-09-15T20:34:41.624 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.AioUnlockPP (3039 ms) 2024-09-15T20:34:41.624 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripAppendPP 2024-09-15T20:35:01.720 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAied=2 2024-09-15T20:35:01.720 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:e960b815:::9:head 2024-09-15T20:35:01.720 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:e960b815:::9:head expected=10:e960b815:::9:head 2024-09-15T20:35:01.720 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:e960b815:::9:head -> 9 2024-09-15T20:35:01.720 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=9 expected=9 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:de5d7c5f:::12:head 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:de5d7c5f:::12:head expected=10:de5d7c5f:::12:head 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:de5d7c5f:::12:head -> 12 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=12 expected=12 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:863748b0:::15:head 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:863748b0:::15:head expected=10:863748b0:::15:head 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:863748b0:::15:head -> 15 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=15 expected=15 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:6cac518f:::0:head 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:6cac518f:::0:head expected=10:6cac518f:::0:head 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:6cac518f:::0:head -> 0 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=0 expected=0 2024-09-15T20:35:01.721 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:62a1935d:::14:head 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:62a1935d:::14:head expected=10:62a1935d:::14:head 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:62a1935d:::14:head -> 14 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=14 expected=14 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:52ea6a34:::10:head 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:52ea6a34:::10:head expected=10:52ea6a34:::10:head 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:52ea6a34:::10:head -> 10 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=10 expected=10 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:566253c9:::13:head 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:566253c9:::13:head expected=10:566253c9:::13:head 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:566253c9:::13:head -> 13 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=13 expected=13 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : seek to 10:02547ec2:::1:head 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : cursor()=10:02547ec2:::1:head expected=10:02547ec2:::1:head 2024-09-15T20:35:01.722 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: > 10:02547ec2:::1:head -> 1 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: : entry=1 expected=1 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosList.ListObjectsCursor (385 ms) 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjects 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosList.EnumerateObjects (103938 ms) 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosList.EnumerateObjectsSplit 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: split 0/5 -> MIN 10:33333333::::head 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: split 1/5 -> 10:33333333::::head 10:66666666::::head 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: split 2/5 -> 10:66666666::::head 10:99999999::::head 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: split 3/5 -> 10:99999999::::head 10:cccccccc::::head 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: split 4/5 -> 10:cccccccc::::head MAX 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosList.EnumerateObjectsSplit (22030 ms) 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [----------] 7 tests from LibRadosList (126871 ms total) 2024-09-15T20:35:01.723 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [----------] 3 tests from LibRadosListEC 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosListEC.ListObjects 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosListEC.ListObjects (1060 ms) 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsNS 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: myset foo1,foo2,foo3 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo1 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo2 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo3 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: myset foo1,foo4,foo5 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo4 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo5 2024-09-15T20:35:01.724 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo1 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: myset foo6,foo7 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo7 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: foo6 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: myset :foo1,:foo2,:foo3,ns1:foo1,ns1:foo4,ns1:foo5,ns2:foo6,ns2:foo7 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns1:foo4 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns1:foo5 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns2:foo7 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns2:foo6 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: ns1:foo1 2024-09-15T20:35:01.725 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: :foo1 2024-09-15T20:35:01.726 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: :foo2 2024-09-15T20:35:01.726 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: :foo3 2024-09-15T20:35:01.726 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsNS (86 ms) 2024-09-15T20:35:07.598 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosListEC.ListObjectsStart 2024-09-15T20:35:07.598 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 1 0 2024-09-15T20:35:07.598 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 10 0 2024-09-15T20:35:07.598 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 13 0 2024-09-15T20:35:07.598 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 7 0 2024-09-15T20:35:07.598 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 14 0 2024-09-15T20:35:07.598 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 0 0 2024-09-15T20:35:07.598 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 15 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 11 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 5 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 8 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 6 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 3 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 4 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 12 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 9 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 2 0 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: have 1 expect one of 0,1,10,11,12,13,14,15,2,3,4,5,6,7,8,9 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosListEC.ListObjectsStart (145 ms) 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [----------] 3 tests from LibRadosListEC (1291 ms total) 2024-09-15T20:35:07.599 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 2024-09-15T20:35:07.600 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [----------] 1 test from LibRadosListNP 2024-09-15T20:35:07.600 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ RUN ] LibRadosListNP.ListObjectsError 2024-09-15T20:35:07.600 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ OK ] LibRadosListNP.ListObjectsError (3033 ms) 2024-09-15T20:35:07.600 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [----------] 1 test from LibRadosListNP (3033 ms total) 2024-09-15T20:35:07.600 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: 2024-09-15T20:35:07.600 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [----------] Global test environment tear-down 2024-09-15T20:35:07.600 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [==========] 11 tests from 3 test suites ran. (138974 ms total) 2024-09-15T20:35:07.600 INFO:tasks.workunit.client.0.smithi146.stdout: api_list: [ PASSED ] 11 tests. 2024-09-15T20:35:28.234 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T20:35:28.235 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T20:35:28.235 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T20:35:28.238 INFO:tasks.ceph.mon.c:Started 2024-09-15T20:35:28.238 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:35:28.239 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient: get_monmap_and_config 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient: build_initial_monmap 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient: monmap: 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:28.365898+0000 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:28.365898+0000 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.366 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:28.367 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:28.367 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 1 Processor -- start 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 1 -- start start 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient: init 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient(hunting): picked mon.noname-a con 0x7fa4bc0689d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient(hunting): picked mon.noname-b con 0x7fa4bc0a0980 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient(hunting): picked mon.noname-c con 0x7fa4bc0694c0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4bc0a1f00 con 0x7fa4bc0689d0 2024-09-15T20:35:28.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4bc09df40 con 0x7fa4bc0694c0 2024-09-15T20:35:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4bc0a2390 con 0x7fa4bc0a0980 2024-09-15T20:35:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c41ce640 10 monclient(hunting): authenticate will time out at 1389.250436s 2024-09-15T20:35:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c1f6a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa4bc0689d0 0x7fa4bc068db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50340/0 (socket says 172.21.15.78:50340) 2024-09-15T20:35:28.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.361+0000 7fa4c1f6a640 1 -- 172.21.15.78:0/3563471267 learned_addr learned my addr 172.21.15.78:0/3563471267 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3857564505 0 0) 0x7fa4bc09df40 con 0x7fa4bc0694c0 2024-09-15T20:35:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient(hunting): my global_id is 23161 2024-09-15T20:35:28.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4a8003530 con 0x7fa4bc0694c0 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2265693476 0 0) 0x7fa4a8003530 con 0x7fa4bc0694c0 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 >> v1:172.21.15.78:6790/0 conn(0x7fa4bc0a0980 legacy=0x7fa4bc13ad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 >> v1:172.21.15.78:6789/0 conn(0x7fa4bc0689d0 legacy=0x7fa4bc068db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: _finish_hunting 0 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 monclient: found mon.noname-c 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4bc0a2560 con 0x7fa4bc0694c0 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: _finish_auth 0 2024-09-15T20:35:28.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: _check_auth_tickets 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa4b4001a20 con 0x7fa4bc0694c0 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: dump: 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.404 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 5 monclient: authenticate success, global_id 23161 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4b4001bc0 con 0x7fa4bc0694c0 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 4 set_mon_vals no callback set 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 1 -- 172.21.15.78:0/3563471267 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa4b4001fd0 con 0x7fa4bc0694c0 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c0f68640 10 monclient: dump: 2024-09-15T20:35:28.405 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 10 monclient: shutdown 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 1 -- 172.21.15.78:0/3563471267 >> v1:172.21.15.146:6789/0 conn(0x7fa4bc0694c0 legacy=0x7fa4bc13e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 1 -- 172.21.15.78:0/3563471267 shutdown_connections 2024-09-15T20:35:28.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 1 -- 172.21.15.78:0/3563471267 >> 172.21.15.78:0/3563471267 conn(0x7fa4bc0a3ce0 msgr2=0x7fa4bc0a40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:28.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 1 -- 172.21.15.78:0/3563471267 shutdown_connections 2024-09-15T20:35:28.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 1 -- 172.21.15.78:0/3563471267 wait complete. 2024-09-15T20:35:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 10 monclient: build_initial_monmap 2024-09-15T20:35:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 10 monclient: monmap: 2024-09-15T20:35:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:28.408 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 1 librados: starting msgr at 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.397+0000 7fa4c41ce640 1 librados: starting objecter 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 1 Processor -- start 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 1 -- start start 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 1 librados: setting wanted keys 2024-09-15T20:35:28.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 1 librados: calling monclient init 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 10 monclient: init 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 10 monclient(hunting): picked mon.noname-c con 0x7fa4bc0689d0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 10 monclient(hunting): picked mon.noname-b con 0x7fa4bc0694c0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 10 monclient(hunting): picked mon.noname-a con 0x7fa4bc0a0980 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4bc133a20 con 0x7fa4bc0a0980 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4bc1cb990 con 0x7fa4bc0694c0 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4bc09e200 con 0x7fa4bc0689d0 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c41ce640 10 monclient(hunting): authenticate will time out at 1389.289207s 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c1f6a640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa4bc0689d0 0x7fa4bc1cab70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53524/0 (socket says 172.21.15.78:53524) 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4c1f6a640 1 -- 172.21.15.78:0/1691920327 learned_addr learned my addr 172.21.15.78:0/1691920327 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:28.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4b27fc640 10 client.?.objecter ms_handle_connect 0x7fa4bc0689d0 2024-09-15T20:35:28.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4b27fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:28.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4b27fc640 10 client.?.objecter ms_handle_connect 0x7fa4bc0694c0 2024-09-15T20:35:28.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4b27fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:28.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4b27fc640 10 client.?.objecter ms_handle_connect 0x7fa4bc0a0980 2024-09-15T20:35:28.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.401+0000 7fa4b27fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3669781613 0 0) 0x7fa4bc133a20 con 0x7fa4bc0a0980 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient(hunting): my global_id is 28581 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa4a0004ec0 con 0x7fa4bc0a0980 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1144713855 0 0) 0x7fa4a0004ec0 con 0x7fa4bc0a0980 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa4bc133a20 con 0x7fa4bc0a0980 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa4b4001f10 con 0x7fa4bc0a0980 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:28.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient(hunting): dump: 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:28.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1848813122 0 0) 0x7fa4bc133a20 con 0x7fa4bc0a0980 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 >> v1:172.21.15.78:6790/0 conn(0x7fa4bc0689d0 legacy=0x7fa4bc1cab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 >> v1:172.21.15.146:6789/0 conn(0x7fa4bc0694c0 legacy=0x7fa4bc1cb280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient: _finish_hunting 0 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 monclient: found mon.a 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4bc1354e0 con 0x7fa4bc0a0980 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient: _finish_auth 0 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient: _check_auth_tickets 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 5 monclient: authenticate success, global_id 28581 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 10 monclient: _renew_subs 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa4bc1357c0 con 0x7fa4bc0a0980 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 10 client.28581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 10 monclient: _renew_subs 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fa4bc1347e0 con 0x7fa4bc0a0980 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4c41ce640 1 librados: init done 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa4b4001b60 con 0x7fa4bc0a0980 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa4b4003e90 con 0x7fa4bc0a0980 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b3fff640 4 set_mon_vals no callback set 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.405+0000 7fa4b27fc640 10 monclient: dump: 2024-09-15T20:35:28.416 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.417 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4c41ce640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4c41ce640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:28.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4c41ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4a4005090 con 0x7fa4bc0a0980 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fa4b400ff00 con 0x7fa4bc0a0980 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(166..166 src has 1..166) ==== 6233+0+0 (unknown 715598834 0 0) 0x7fa4b404ac20 con 0x7fa4bc0a0980 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4b27fc640 10 client.28581.objecter ms_dispatch 0x7fa4bc000ba0 osd_map(166..166 src has 1..166) 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4b27fc640 3 client.28581.objecter handle_osd_map got epochs [166,166] > 0 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4b27fc640 3 client.28581.objecter handle_osd_map decoding full epoch 166 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4b27fc640 20 client.28581.objecter dump_active .. 0 homeless 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4c1f6a640 10 monclient: get_auth_request con 0x7fa4a003dfe0 auth_method 0 2024-09-15T20:35:28.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.409+0000 7fa4b27fc640 10 client.28581.objecter ms_handle_connect 0x7fa4a003dfe0 2024-09-15T20:35:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.413+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fa4b4015b50 con 0x7fa4bc0a0980 2024-09-15T20:35:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.413+0000 7fa4b27fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:28.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.413+0000 7fa4b27fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:28.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.677+0000 7fa4c41ce640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:35:28.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.677+0000 7fa4c41ce640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:28.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.677+0000 7fa4c41ce640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.677+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa4a4005380 con 0x7fa4bc0a0980 2024-09-15T20:35:28.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.677+0000 7fa4b27fc640 1 -- 172.21.15.78:0/1691920327 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3389542080) 0x7fa4b4019ed0 con 0x7fa4bc0a0980 2024-09-15T20:35:28.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.677+0000 7fa4b27fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:28.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.677+0000 7fa4b27fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:28.684 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:35:28.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 10 librados: watch_flush enter 2024-09-15T20:35:28.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 10 librados: watch_flush exit 2024-09-15T20:35:28.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 20 client.28581.objecter shutdown clearing up homeless session... 2024-09-15T20:35:28.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 10 client.28581.objecter successfully canceled tick 2024-09-15T20:35:28.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 >> v1:172.21.15.146:6812/435598642 conn(0x7fa4a003dfe0 legacy=0x7fa4a00404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 10 monclient: shutdown 2024-09-15T20:35:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 >> v1:172.21.15.78:6789/0 conn(0x7fa4bc0a0980 legacy=0x7fa4bc1d2dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 shutdown_connections 2024-09-15T20:35:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 >> 172.21.15.78:0/1691920327 conn(0x7fa4bc0a3ce0 msgr2=0x7fa4bc0a5ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 shutdown_connections 2024-09-15T20:35:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 1 -- 172.21.15.78:0/1691920327 wait complete. 2024-09-15T20:35:28.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.681+0000 7fa4c41ce640 1 librados: shutdown 2024-09-15T20:35:28.702 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:35:28.702 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:35:28.702 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:35:28.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient: get_monmap_and_config 2024-09-15T20:35:28.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient: build_initial_monmap 2024-09-15T20:35:28.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient: monmap: 2024-09-15T20:35:28.821 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:28.821296+0000 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:28.821296+0000 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:28.822 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 1 Processor -- start 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 1 -- start start 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient: init 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient(hunting): picked mon.noname-c con 0x7f214c134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient(hunting): picked mon.noname-a con 0x7f214c09f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient(hunting): picked mon.noname-b con 0x7f214c06e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f214c0a0470 con 0x7f214c09f5a0 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f214c09feb0 con 0x7f214c134bb0 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f214c09e4a0 con 0x7f214c06e960 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:28.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient(hunting): authenticate will time out at 1389.705876s 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f214c09f5a0 0x7f214c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50360/0 (socket says 172.21.15.78:50360) 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f2150c63640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f214c06e960 0x7f214c13d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53526/0 (socket says 172.21.15.78:53526) 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214b7fe640 1 -- 172.21.15.78:0/3612050480 learned_addr learned my addr 172.21.15.78:0/3612050480 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025694719 0 0) 0x7f213c001120 con 0x7f214c09f5a0 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient(hunting): my global_id is 28584 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2138003530 con 0x7f214c09f5a0 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1416174172 0 0) 0x7f214c09feb0 con 0x7f214c134bb0 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient(hunting): my global_id is 23164 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f213c001120 con 0x7f214c134bb0 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3829562112 0 0) 0x7f2138003530 con 0x7f214c09f5a0 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 >> v1:172.21.15.78:6790/0 conn(0x7f214c06e960 legacy=0x7f214c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 >> v1:172.21.15.146:6789/0 conn(0x7f214c134bb0 legacy=0x7f214c134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: _finish_hunting 0 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 monclient: found mon.noname-a 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214c0a0ff0 con 0x7f214c09f5a0 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: _finish_auth 0 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: _check_auth_tickets 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:28.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f213c001870 con 0x7f214c09f5a0 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: dump: 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.829 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 5 monclient: authenticate success, global_id 28584 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f213c001a90 con 0x7f214c09f5a0 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 1 -- 172.21.15.78:0/3612050480 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f213c0048b0 con 0x7f214c09f5a0 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 4 set_mon_vals no callback set 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f214affd640 10 monclient: dump: 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.830 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 10 monclient: shutdown 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:28.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.817+0000 7f21526c6640 1 -- 172.21.15.78:0/3612050480 >> v1:172.21.15.78:6789/0 conn(0x7f214c09f5a0 legacy=0x7f214c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- 172.21.15.78:0/3612050480 shutdown_connections 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- 172.21.15.78:0/3612050480 >> 172.21.15.78:0/3612050480 conn(0x7f214c0a63c0 msgr2=0x7f214c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- 172.21.15.78:0/3612050480 shutdown_connections 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- 172.21.15.78:0/3612050480 wait complete. 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: build_initial_monmap 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: monmap: 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:28.833 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 librados: starting msgr at 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 librados: starting objecter 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 Processor -- start 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- start start 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 librados: setting wanted keys 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 librados: calling monclient init 2024-09-15T20:35:28.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: init 2024-09-15T20:35:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:35:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient(hunting): picked mon.noname-a con 0x7f214c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient(hunting): picked mon.noname-b con 0x7f214c09f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:28.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient(hunting): picked mon.noname-c con 0x7f214c134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f214c123a60 con 0x7f214c06e960 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f214c1218d0 con 0x7f214c09f5a0 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f214c122570 con 0x7f214c134bb0 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient(hunting): authenticate will time out at 1389.709690s 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2150c63640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f214c134bb0 0x7f214c1d5110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53528/0 (socket says 172.21.15.78:53528) 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2150c63640 1 -- 172.21.15.78:0/1460894142 learned_addr learned my addr 172.21.15.78:0/1460894142 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 client.?.objecter ms_handle_connect 0x7f214c06e960 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 client.?.objecter ms_handle_connect 0x7f214c134bb0 2024-09-15T20:35:28.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 client.?.objecter ms_handle_connect 0x7f214c09f5a0 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1381904896 0 0) 0x7f214c123a60 con 0x7f214c06e960 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): my global_id is 28587 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f212c004ec0 con 0x7f214c06e960 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2455073478 0 0) 0x7f214c122570 con 0x7f214c134bb0 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): my global_id is 84101 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f214c123a60 con 0x7f214c134bb0 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4122527251 0 0) 0x7f212c004ec0 con 0x7f214c06e960 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f214c122570 con 0x7f214c06e960 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2659152036 0 0) 0x7f214c123a60 con 0x7f214c134bb0 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f212c004ec0 con 0x7f214c134bb0 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2134001940 con 0x7f214c06e960 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): dump: 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.843 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2144001bc0 con 0x7f214c134bb0 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient(hunting): dump: 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.844 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2293809630 0 0) 0x7f214c122570 con 0x7f214c06e960 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 >> v1:172.21.15.78:6790/0 conn(0x7f214c134bb0 legacy=0x7f214c1d5110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 >> v1:172.21.15.146:6789/0 conn(0x7f214c09f5a0 legacy=0x7f214c1211c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:35:28.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 monclient: found mon.a 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f214c1248d0 con 0x7f214c06e960 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient: _finish_auth 0 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 5 monclient: authenticate success, global_id 28587 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: _renew_subs 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f214c133200 con 0x7f214c06e960 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 client.28587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: _renew_subs 2024-09-15T20:35:28.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f214c1230d0 con 0x7f214c06e960 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f21526c6640 1 librados: init done 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2134001ca0 con 0x7f214c06e960 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f21340038d0 con 0x7f214c06e960 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f214a7fc640 4 set_mon_vals no callback set 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.821+0000 7f2148ff9640 10 monclient: dump: 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.847 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f21526c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f21300065d0 con 0x7f2130002870 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 client.28587.objecter ms_handle_connect 0x7f2130002870 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 client.28587.objecter resend_mon_ops 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3579788 0 0) 0x7f21300065d0 con 0x7f2130002870 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient(hunting): my global_id is 28590 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f214c122570 con 0x7f2130002870 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f213400f9c0 con 0x7f214c06e960 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(167..167 src has 1..167) ==== 6193+0+0 (unknown 293837830 0 0) 0x7f213404a9a0 con 0x7f214c06e960 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 client.28587.objecter ms_dispatch 0x7f214c000ba0 osd_map(167..167 src has 1..167) 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 3 client.28587.objecter handle_osd_map got epochs [167,167] > 0 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 3 client.28587.objecter handle_osd_map decoding full epoch 167 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 20 client.28587.objecter dump_active .. 0 homeless 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1038550501 0 0) 0x7f214c122570 con 0x7f2130002870 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f21300065d0 con 0x7f2130002870 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f213c001610 con 0x7f2130002870 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient: dump: 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 654781395 0 0) 0x7f21300065d0 con 0x7f2130002870 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f213000a170 con 0x7f2130002870 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2150c63640 10 monclient: get_auth_request con 0x7f212c03e150 auth_method 0 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f213c004b70 con 0x7f2130002870 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 >> v1:172.21.15.78:6789/0 conn(0x7f2130002870 legacy=0x7f2130004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 >> v1:172.21.15.78:6789/0 conn(0x7f2130002870 legacy=0x7f2130004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.825+0000 7f2148ff9640 10 client.28587.objecter ms_handle_connect 0x7f212c03e150 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f21526c6640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2130009e80 con 0x7f2130009140 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f2148ff9640 10 client.28587.objecter ms_handle_connect 0x7f2130009140 2024-09-15T20:35:28.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f2148ff9640 10 client.28587.objecter resend_mon_ops 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3057269121 0 0) 0x7f2130009e80 con 0x7f2130009140 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f2148ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f2148ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f2148ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f2148ff9640 10 monclient(hunting): my global_id is 28593 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.829+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f21300065d0 con 0x7f2130009140 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 343240795 0 0) 0x7f21300065d0 con 0x7f2130009140 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2130009e80 con 0x7f2130009140 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f213c001b20 con 0x7f2130009140 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:28.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 10 monclient: dump: 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1590456219 0 0) 0x7f2130009e80 con 0x7f2130009140 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f213000c420 con 0x7f2130009140 2024-09-15T20:35:28.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4140 (unknown 0 0 3579331726) 0x7f213c001a20 con 0x7f2130009140 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 >> v1:172.21.15.78:6789/0 conn(0x7f2130009140 legacy=0x7f21300095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2148ff9640 1 -- 172.21.15.78:0/1460894142 >> v1:172.21.15.78:6789/0 conn(0x7f2130009140 legacy=0x7f21300095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 16, 2024-09-15T20:35:28.854 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 241384, 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:28.855 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:28.856 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:35:28.857 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:35:28.858 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.859 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:35:28.860 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:35:28.861 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:28.862 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:28.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:28.864 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:35:28.865 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:35:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 10 librados: watch_flush enter 2024-09-15T20:35:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 10 librados: watch_flush exit 2024-09-15T20:35:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 20 client.28587.objecter shutdown clearing up homeless session... 2024-09-15T20:35:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 10 client.28587.objecter successfully canceled tick 2024-09-15T20:35:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 >> v1:172.21.15.146:6812/435598642 conn(0x7f212c03e150 legacy=0x7f212c040610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 10 monclient: shutdown 2024-09-15T20:35:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:28.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 >> v1:172.21.15.78:6789/0 conn(0x7f214c06e960 legacy=0x7f214c120ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2150c63640 1 -- 172.21.15.78:0/1460894142 reap_dead start 2024-09-15T20:35:28.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f2150c63640 1 -- 172.21.15.78:0/1460894142 reap_dead start 2024-09-15T20:35:28.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 shutdown_connections 2024-09-15T20:35:28.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 >> 172.21.15.78:0/1460894142 conn(0x7f214c0a63c0 msgr2=0x7f214c0a2a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:28.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 shutdown_connections 2024-09-15T20:35:28.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 1 -- 172.21.15.78:0/1460894142 wait complete. 2024-09-15T20:35:28.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.833+0000 7f21526c6640 1 librados: shutdown 2024-09-15T20:35:28.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:35:28.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:35:28.869 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:35:28.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient: get_monmap_and_config 2024-09-15T20:35:28.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient: build_initial_monmap 2024-09-15T20:35:28.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient: monmap: 2024-09-15T20:35:28.982 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:28.982 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.982 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:28.981887+0000 2024-09-15T20:35:28.982 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:28.981887+0000 2024-09-15T20:35:28.983 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:28.983 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.983 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:28.983 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:28.983 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:28.983 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 1 Processor -- start 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 1 -- start start 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient: init 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient(hunting): picked mon.noname-b con 0x7facb4137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient(hunting): picked mon.noname-a con 0x7facb4074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient(hunting): picked mon.noname-c con 0x7facb413e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb4138850 con 0x7facb4074500 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb40a5da0 con 0x7facb413e370 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb4073560 con 0x7facb4137d60 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facbab22640 10 monclient(hunting): authenticate will time out at 1389.867310s 2024-09-15T20:35:28.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7facb4074500 0x7facb413cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50394/0 (socket says 172.21.15.78:50394) 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb3fff640 1 -- 172.21.15.78:0/1286590569 learned_addr learned my addr 172.21.15.78:0/1286590569 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 7503554 0 0) 0x7facb4138850 con 0x7facb4074500 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 10 monclient(hunting): my global_id is 28596 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faca0003530 con 0x7facb4074500 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1012109329 0 0) 0x7facb4073560 con 0x7facb4137d60 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 10 monclient(hunting): my global_id is 84104 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.977+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facb4138850 con 0x7facb4137d60 2024-09-15T20:35:28.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 729173020 0 0) 0x7faca0003530 con 0x7facb4074500 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 >> v1:172.21.15.78:6790/0 conn(0x7facb4137d60 legacy=0x7facb4138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 >> v1:172.21.15.146:6789/0 conn(0x7facb413e370 legacy=0x7facb4140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: _finish_hunting 0 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 1 monclient: found mon.noname-a 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb4073840 con 0x7facb4074500 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: _finish_auth 0 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: _check_auth_tickets 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fac9c0016a0 con 0x7facb4074500 2024-09-15T20:35:28.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 5 monclient: authenticate success, global_id 28596 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: dump: 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.990 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac9c0018c0 con 0x7facb4074500 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 1 -- 172.21.15.78:0/1286590569 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fac9c001cd0 con 0x7facb4074500 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb37fe640 10 monclient: dump: 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:28.991 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 4 set_mon_vals no callback set 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: shutdown 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- 172.21.15.78:0/1286590569 >> v1:172.21.15.78:6789/0 conn(0x7facb4074500 legacy=0x7facb413cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- 172.21.15.78:0/1286590569 shutdown_connections 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- 172.21.15.78:0/1286590569 >> 172.21.15.78:0/1286590569 conn(0x7facb40aa650 msgr2=0x7facb40aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- 172.21.15.78:0/1286590569 shutdown_connections 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- 172.21.15.78:0/1286590569 wait complete. 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: build_initial_monmap 2024-09-15T20:35:28.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: monmap: 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:28.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 librados: starting msgr at 2024-09-15T20:35:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 librados: starting objecter 2024-09-15T20:35:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 Processor -- start 2024-09-15T20:35:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- start start 2024-09-15T20:35:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 librados: setting wanted keys 2024-09-15T20:35:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 librados: calling monclient init 2024-09-15T20:35:28.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: init 2024-09-15T20:35:28.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:28.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:35:28.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient(hunting): picked mon.noname-a con 0x7facb4074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:28.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient(hunting): picked mon.noname-c con 0x7facb4137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:28.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient(hunting): picked mon.noname-b con 0x7facb413e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:28.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb4141960 con 0x7facb4074500 2024-09-15T20:35:28.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb4135c60 con 0x7facb413e370 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7facb4133390 con 0x7facb4137d60 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient(hunting): authenticate will time out at 1389.870549s 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb88be640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7facb4074500 0x7facb4137190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50400/0 (socket says 172.21.15.78:50400) 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb88be640 1 -- 172.21.15.78:0/3434829100 learned_addr learned my addr 172.21.15.78:0/3434829100 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 client.?.objecter ms_handle_connect 0x7facb4137d60 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 client.?.objecter ms_handle_connect 0x7facb4074500 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1428856441 0 0) 0x7facb4133390 con 0x7facb4137d60 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): my global_id is 84107 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fac94004ef0 con 0x7facb4137d60 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1249405292 0 0) 0x7faca4001c80 con 0x7facb4074500 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): my global_id is 28599 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facb4133390 con 0x7facb4074500 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1290510734 0 0) 0x7fac94004ef0 con 0x7facb4137d60 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faca4001c80 con 0x7facb4137d60 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fac9c001c10 con 0x7facb4137d60 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:29.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): dump: 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:29.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 client.?.objecter ms_handle_connect 0x7facb413e370 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4191569549 0 0) 0x7facb4133390 con 0x7facb4074500 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fac94004ef0 con 0x7facb4074500 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faca4001f90 con 0x7facb4074500 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient(hunting): dump: 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.003 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3314936241 0 0) 0x7faca4001c80 con 0x7facb4137d60 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 >> v1:172.21.15.146:6789/0 conn(0x7facb413e370 legacy=0x7facb41d4df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 >> v1:172.21.15.78:6789/0 conn(0x7facb4074500 legacy=0x7facb4137190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient: _finish_hunting 0 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 monclient: found mon.c 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb409e0b0 con 0x7facb4137d60 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient: _finish_auth 0 2024-09-15T20:35:29.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 10 monclient: _check_auth_tickets 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facb17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 5 monclient: authenticate success, global_id 84107 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: _renew_subs 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7facb4135980 con 0x7facb4137d60 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 client.84107.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: _renew_subs 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7facb41378a0 con 0x7facb4137d60 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 1 librados: init done 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.981+0000 7facbab22640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7facb4134580 con 0x7facb41c8de0 2024-09-15T20:35:29.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fac9c001170 con 0x7facb4137d60 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fac9c004c00 con 0x7facb4137d60 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: dump: 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.006 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb2ffd640 4 set_mon_vals no callback set 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 client.84107.objecter ms_handle_connect 0x7facb41c8de0 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 client.84107.objecter resend_mon_ops 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fac9c010c70 con 0x7facb4137d60 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(167..167 src has 1..167) ==== 6193+0+0 (unknown 293837830 0 0) 0x7fac9c04ba60 con 0x7facb4137d60 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 client.84107.objecter ms_dispatch 0x7facb4000ba0 osd_map(167..167 src has 1..167) 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 3 client.84107.objecter handle_osd_map got epochs [167,167] > 0 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 3 client.84107.objecter handle_osd_map decoding full epoch 167 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 20 client.84107.objecter dump_active .. 0 homeless 2024-09-15T20:35:29.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 890828603 0 0) 0x7facb4134580 con 0x7facb41c8de0 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient(hunting): my global_id is 84110 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faca4001c80 con 0x7facb41c8de0 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb90bf640 10 monclient: get_auth_request con 0x7fac9403e070 auth_method 0 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1322625396 0 0) 0x7faca4001c80 con 0x7facb41c8de0 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7facb4134580 con 0x7facb41c8de0 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faca4003d90 con 0x7facb41c8de0 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:29.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: dump: 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 7106323 0 0) 0x7facb4134580 con 0x7facb41c8de0 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7facb41c91c0 con 0x7facb41c8de0 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 client.84107.objecter ms_handle_connect 0x7fac9403e070 2024-09-15T20:35:29.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7faca4005b10 con 0x7facb41c8de0 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 >> v1:172.21.15.78:6790/0 conn(0x7facb41c8de0 legacy=0x7facb40a2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.985+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 >> v1:172.21.15.78:6790/0 conn(0x7facb41c8de0 legacy=0x7facb40a2c00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.989+0000 7facbab22640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.989+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7facb4076db0 con 0x7facb402e460 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 client.84107.objecter ms_handle_connect 0x7facb402e460 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 client.84107.objecter resend_mon_ops 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3617725396 0 0) 0x7facb4076db0 con 0x7facb402e460 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient(hunting): my global_id is 84113 2024-09-15T20:35:29.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7facb4134580 con 0x7facb402e460 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3228153723 0 0) 0x7facb4134580 con 0x7facb402e460 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7facb4076db0 con 0x7facb402e460 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faca4004480 con 0x7facb402e460 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient: dump: 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.011 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 247685751 0 0) 0x7facb4076db0 con 0x7facb402e460 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7facb409cfc0 con 0x7facb402e460 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 2368319747) 0x7faca40044c0 con 0x7facb402e460 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 >> v1:172.21.15.78:6790/0 conn(0x7facb402e460 legacy=0x7facb40a3310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb17fa640 1 -- 172.21.15.78:0/3434829100 >> v1:172.21.15.78:6790/0 conn(0x7facb402e460 legacy=0x7facb40a3310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 16, 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 679, 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:35:29.013 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:29.014 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:35:29.016 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:29.017 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:29.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.019 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.020 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:29.021 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.022 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:35:29.023 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:35:29.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 10 librados: watch_flush enter 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 10 librados: watch_flush exit 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 20 client.84107.objecter shutdown clearing up homeless session... 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 10 client.84107.objecter successfully canceled tick 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 >> v1:172.21.15.146:6812/435598642 conn(0x7fac9403e070 legacy=0x7fac94040530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 10 monclient: shutdown 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 >> v1:172.21.15.78:6790/0 conn(0x7facb4137d60 legacy=0x7facb41d16c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb90bf640 1 -- 172.21.15.78:0/3434829100 reap_dead start 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facb90bf640 1 -- 172.21.15.78:0/3434829100 reap_dead start 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 shutdown_connections 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 >> 172.21.15.78:0/3434829100 conn(0x7facb40aa650 msgr2=0x7facb4140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 shutdown_connections 2024-09-15T20:35:29.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 1 -- 172.21.15.78:0/3434829100 wait complete. 2024-09-15T20:35:29.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:28.993+0000 7facbab22640 1 librados: shutdown 2024-09-15T20:35:29.026 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:35:29.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:35:29.026 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:35:29.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient: get_monmap_and_config 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient: build_initial_monmap 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient: monmap: 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:29.145397+0000 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:29.145397+0000 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:29.146 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 Processor -- start 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- start start 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient: init 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c0745e0 con 0x7fcf1c13c220 2024-09-15T20:35:29.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c06ecf0 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c06efd0 con 0x7fcf1c135b60 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient(hunting): authenticate will time out at 1390.029967s 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf21c00640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcf1c135b60 0x7fcf1c135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53582/0 (socket says 172.21.15.78:53582) 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf21c00640 1 -- 172.21.15.78:0/3133259651 learned_addr learned my addr 172.21.15.78:0/3133259651 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 296073785 0 0) 0x7fcf1c06efd0 con 0x7fcf1c135b60 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient(hunting): my global_id is 84116 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf08003530 con 0x7fcf1c135b60 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3636221935 0 0) 0x7fcf1c0745e0 con 0x7fcf1c13c220 2024-09-15T20:35:29.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient(hunting): my global_id is 28602 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf1c06efd0 con 0x7fcf1c13c220 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 588691941 0 0) 0x7fcf08003530 con 0x7fcf1c135b60 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 >> v1:172.21.15.146:6789/0 conn(0x7fcf1c0a24e0 legacy=0x7fcf1c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 >> v1:172.21.15.78:6789/0 conn(0x7fcf1c13c220 legacy=0x7fcf1c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: _finish_hunting 0 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 monclient: found mon.noname-b 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c136650 con 0x7fcf1c135b60 2024-09-15T20:35:29.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: _finish_auth 0 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: _check_auth_tickets 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcf0c0016a0 con 0x7fcf1c135b60 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 5 monclient: authenticate success, global_id 84116 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: dump: 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.158 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf0c0018c0 con 0x7fcf1c135b60 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 1 -- 172.21.15.78:0/3133259651 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcf0c001cd0 con 0x7fcf1c135b60 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf20bfe640 10 monclient: dump: 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.159 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 4 set_mon_vals no callback set 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 10 monclient: shutdown 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- 172.21.15.78:0/3133259651 >> v1:172.21.15.78:6790/0 conn(0x7fcf1c135b60 legacy=0x7fcf1c135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- 172.21.15.78:0/3133259651 shutdown_connections 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- 172.21.15.78:0/3133259651 >> 172.21.15.78:0/3133259651 conn(0x7fcf1c0a63c0 msgr2=0x7fcf1c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- 172.21.15.78:0/3133259651 shutdown_connections 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.141+0000 7fcf23e64640 1 -- 172.21.15.78:0/3133259651 wait complete. 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: build_initial_monmap 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: monmap: 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:29.161 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 librados: starting msgr at 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 librados: starting objecter 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 Processor -- start 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 -- start start 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 librados: setting wanted keys 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 librados: calling monclient init 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: init 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:29.162 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient(hunting): picked mon.noname-c con 0x7fcf1c0a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient(hunting): picked mon.noname-b con 0x7fcf1c135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient(hunting): picked mon.noname-a con 0x7fcf1c13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c138080 con 0x7fcf1c13c220 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c138360 con 0x7fcf1c135b60 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf1c072c90 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient(hunting): authenticate will time out at 1390.033514s 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf22401640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcf1c13c220 0x7fcf1c1d2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50408/0 (socket says 172.21.15.78:50408) 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf21c00640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcf1c0a24e0 0x7fcf1c071e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53586/0 (socket says 172.21.15.78:53586) 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf22401640 1 -- 172.21.15.78:0/2345333634 learned_addr learned my addr 172.21.15.78:0/2345333634 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.?.objecter ms_handle_connect 0x7fcf1c13c220 2024-09-15T20:35:29.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.?.objecter ms_handle_connect 0x7fcf1c0a24e0 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405647671 0 0) 0x7fcf1c072c90 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): my global_id is 84119 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf00004ec0 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346801955 0 0) 0x7fcf1c138080 con 0x7fcf1c13c220 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): my global_id is 28605 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf1c072c90 con 0x7fcf1c13c220 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.?.objecter ms_handle_connect 0x7fcf1c135b60 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 566251477 0 0) 0x7fcf00004ec0 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf1c138080 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcf0c001c10 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): dump: 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.166 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3521848695 0 0) 0x7fcf1c072c90 con 0x7fcf1c13c220 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf00004ec0 con 0x7fcf1c13c220 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcf140016a0 con 0x7fcf1c13c220 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): dump: 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.167 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2104280729 0 0) 0x7fcf1c138080 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 >> v1:172.21.15.146:6789/0 conn(0x7fcf1c135b60 legacy=0x7fcf1c072580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 >> v1:172.21.15.78:6789/0 conn(0x7fcf1c13c220 legacy=0x7fcf1c1d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient: _finish_hunting 0 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 monclient: found mon.c 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf1c138920 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient: _finish_auth 0 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient: _check_auth_tickets 2024-09-15T20:35:29.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 5 monclient: authenticate success, global_id 84119 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: _renew_subs 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcf1c13bb20 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 client.84119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: _renew_subs 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcf1c138640 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 librados: init done 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcf0c001820 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf13fff640 4 set_mon_vals no callback set 2024-09-15T20:35:29.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcf0c004c00 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient: dump: 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.170 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcf040065d0 con 0x7fcf04002870 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fcf0c010c70 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.84119.objecter ms_handle_connect 0x7fcf04002870 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.84119.objecter resend_mon_ops 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(167..167 src has 1..167) ==== 6193+0+0 (unknown 293837830 0 0) 0x7fcf0c04bab0 con 0x7fcf1c0a24e0 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 client.84119.objecter ms_dispatch 0x7fcf1c000ba0 osd_map(167..167 src has 1..167) 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 3 client.84119.objecter handle_osd_map got epochs [167,167] > 0 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 3 client.84119.objecter handle_osd_map decoding full epoch 167 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 20 client.84119.objecter dump_active .. 0 homeless 2024-09-15T20:35:29.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2990475751 0 0) 0x7fcf040065d0 con 0x7fcf04002870 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 10 monclient(hunting): my global_id is 23179 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf1c138080 con 0x7fcf04002870 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.145+0000 7fcf22401640 10 monclient: get_auth_request con 0x7fcf0003e040 auth_method 0 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 10 client.84119.objecter ms_handle_connect 0x7fcf0003e040 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1265959498 0 0) 0x7fcf1c138080 con 0x7fcf04002870 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf040065d0 con 0x7fcf04002870 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcf180018a0 con 0x7fcf04002870 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 10 monclient: dump: 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3449944495 0 0) 0x7fcf040065d0 con 0x7fcf04002870 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf0400a170 con 0x7fcf04002870 2024-09-15T20:35:29.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcf18001960 con 0x7fcf04002870 2024-09-15T20:35:29.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:29.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:29.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 >> v1:172.21.15.146:6789/0 conn(0x7fcf04002870 legacy=0x7fcf04004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.149+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 >> v1:172.21.15.146:6789/0 conn(0x7fcf04002870 legacy=0x7fcf04004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:29.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.153+0000 7fcf23e64640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:35:29.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.153+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcf04009e80 con 0x7fcf04009140 2024-09-15T20:35:29.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 client.84119.objecter ms_handle_connect 0x7fcf04009140 2024-09-15T20:35:29.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 client.84119.objecter resend_mon_ops 2024-09-15T20:35:29.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3810311638 0 0) 0x7fcf04009e80 con 0x7fcf04009140 2024-09-15T20:35:29.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient(hunting): my global_id is 23182 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf040065d0 con 0x7fcf04009140 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4151816111 0 0) 0x7fcf040065d0 con 0x7fcf04009140 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcf04009e80 con 0x7fcf04009140 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcf18003820 con 0x7fcf04009140 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient: dump: 2024-09-15T20:35:29.176 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1255138414 0 0) 0x7fcf04009e80 con 0x7fcf04009140 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf0400c420 con 0x7fcf04009140 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 1296164920) 0x7fcf18005aa0 con 0x7fcf04009140 2024-09-15T20:35:29.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 >> v1:172.21.15.146:6789/0 conn(0x7fcf04009140 legacy=0x7fcf040095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf127fc640 1 -- 172.21.15.78:0/2345333634 >> v1:172.21.15.146:6789/0 conn(0x7fcf04009140 legacy=0x7fcf040095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 16, 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 241669, 2024-09-15T20:35:29.178 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:29.179 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:29.180 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:35:29.181 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:35:29.182 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:35:29.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.184 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:29.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:29.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:35:29.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:35:29.189 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 7 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:29.190 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:35:29.191 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:35:29.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 10 librados: watch_flush enter 2024-09-15T20:35:29.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 10 librados: watch_flush exit 2024-09-15T20:35:29.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 20 client.84119.objecter shutdown clearing up homeless session... 2024-09-15T20:35:29.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 10 client.84119.objecter successfully canceled tick 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 >> v1:172.21.15.146:6812/435598642 conn(0x7fcf0003e040 legacy=0x7fcf00040500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 10 monclient: shutdown 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 >> v1:172.21.15.78:6790/0 conn(0x7fcf1c0a24e0 legacy=0x7fcf1c071e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf22401640 1 -- 172.21.15.78:0/2345333634 reap_dead start 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf22401640 1 -- 172.21.15.78:0/2345333634 reap_dead start 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 shutdown_connections 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 >> 172.21.15.78:0/2345333634 conn(0x7fcf1c0a63c0 msgr2=0x7fcf1c12e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 shutdown_connections 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 1 -- 172.21.15.78:0/2345333634 wait complete. 2024-09-15T20:35:29.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.157+0000 7fcf23e64640 1 librados: shutdown 2024-09-15T20:35:29.194 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:35:29.194 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:35:29.194 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:35:29.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient: get_monmap_and_config 2024-09-15T20:35:29.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient: build_initial_monmap 2024-09-15T20:35:29.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient: monmap: 2024-09-15T20:35:29.306 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:29.306250+0000 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:29.306250+0000 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:29.307 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 1 Processor -- start 2024-09-15T20:35:29.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 1 -- start start 2024-09-15T20:35:29.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient: init 2024-09-15T20:35:29.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:29.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:29.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient(hunting): picked mon.noname-b con 0x7f8754134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient(hunting): picked mon.noname-c con 0x7f875409f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient(hunting): picked mon.noname-a con 0x7f875406e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87540a0470 con 0x7f875406e960 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f875409feb0 con 0x7f875409f5a0 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f875409e4a0 con 0x7f8754134bb0 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8759a1b640 10 monclient(hunting): authenticate will time out at 1390.191332s 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87537fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f875406e960 0x7f875413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50416/0 (socket says 172.21.15.78:50416) 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f8752ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f8754134bb0 0x7f8754134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53598/0 (socket says 172.21.15.78:53598) 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87537fe640 1 -- 172.21.15.78:0/4261847685 learned_addr learned my addr 172.21.15.78:0/4261847685 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3616243116 0 0) 0x7f874c0014a0 con 0x7f875406e960 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 10 monclient(hunting): my global_id is 28608 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8740003530 con 0x7f875406e960 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1346619831 0 0) 0x7f875409e4a0 con 0x7f8754134bb0 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 10 monclient(hunting): my global_id is 84122 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.301+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f874c0014a0 con 0x7f8754134bb0 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4198721939 0 0) 0x7f874c0014a0 con 0x7f8754134bb0 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 >> v1:172.21.15.146:6789/0 conn(0x7f875409f5a0 legacy=0x7f87541399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 >> v1:172.21.15.78:6789/0 conn(0x7f875406e960 legacy=0x7f875413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: _finish_hunting 0 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 1 monclient: found mon.noname-b 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87540a0ff0 con 0x7f8754134bb0 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: _finish_auth 0 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: _check_auth_tickets 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8744001680 con 0x7f8754134bb0 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: dump: 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.313 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 5 monclient: authenticate success, global_id 84122 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87440018a0 con 0x7f8754134bb0 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 1 -- 172.21.15.78:0/4261847685 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8744001cb0 con 0x7f8754134bb0 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:29.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 4 set_mon_vals no callback set 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f87527fc640 10 monclient: dump: 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: shutdown 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- 172.21.15.78:0/4261847685 >> v1:172.21.15.78:6790/0 conn(0x7f8754134bb0 legacy=0x7f8754134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- 172.21.15.78:0/4261847685 shutdown_connections 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- 172.21.15.78:0/4261847685 >> 172.21.15.78:0/4261847685 conn(0x7f87540a63c0 msgr2=0x7f87540a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- 172.21.15.78:0/4261847685 shutdown_connections 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- 172.21.15.78:0/4261847685 wait complete. 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: build_initial_monmap 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: monmap: 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:29.316 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 librados: starting msgr at 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 librados: starting objecter 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 Processor -- start 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- start start 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 librados: setting wanted keys 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 librados: calling monclient init 2024-09-15T20:35:29.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: init 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient(hunting): picked mon.noname-a con 0x7f875406e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient(hunting): picked mon.noname-c con 0x7f875409f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient(hunting): picked mon.noname-b con 0x7f8754134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87540a4610 con 0x7f875406e960 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f875409d180 con 0x7f8754134bb0 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8754121fa0 con 0x7f875409f5a0 2024-09-15T20:35:29.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient(hunting): authenticate will time out at 1390.194214s 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8752ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f875406e960 0x7f8754133910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50430/0 (socket says 172.21.15.78:50430) 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8752ffd640 1 -- 172.21.15.78:0/221872895 learned_addr learned my addr 172.21.15.78:0/221872895 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f874a7fc640 1 --1- 172.21.15.78:0/221872895 >> v1:172.21.15.78:6790/0 conn(0x7f875409f5a0 0x7f87541cd8e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53604/0 (socket says 172.21.15.78:53604) 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 client.?.objecter ms_handle_connect 0x7f875406e960 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 client.?.objecter ms_handle_connect 0x7f875409f5a0 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2507680394 0 0) 0x7f8754121fa0 con 0x7f875409f5a0 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): my global_id is 84125 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8734004ec0 con 0x7f875409f5a0 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3909120663 0 0) 0x7f87540a0ff0 con 0x7f875406e960 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): my global_id is 28611 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8754121fa0 con 0x7f875406e960 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 client.?.objecter ms_handle_connect 0x7f8754134bb0 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4249988228 0 0) 0x7f8734004ec0 con 0x7f875409f5a0 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87540a0ff0 con 0x7f875409f5a0 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f873c001980 con 0x7f875409f5a0 2024-09-15T20:35:29.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): dump: 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 479064727 0 0) 0x7f8754121fa0 con 0x7f875406e960 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8734004ec0 con 0x7f875406e960 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8744001bf0 con 0x7f875406e960 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): dump: 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.325 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3348276824 0 0) 0x7f875409d180 con 0x7f8754134bb0 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient(hunting): my global_id is 23188 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8754121fa0 con 0x7f8754134bb0 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1433446819 0 0) 0x7f87540a0ff0 con 0x7f875409f5a0 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 >> v1:172.21.15.146:6789/0 conn(0x7f8754134bb0 legacy=0x7f87541d1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 >> v1:172.21.15.78:6789/0 conn(0x7f875406e960 legacy=0x7f8754133910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:35:29.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 monclient: found mon.c 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8754124020 con 0x7f875409f5a0 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient: _finish_auth 0 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 5 monclient: authenticate success, global_id 84125 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: _renew_subs 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8754124300 con 0x7f875409f5a0 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 client.84125.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: _renew_subs 2024-09-15T20:35:29.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f87541c8800 con 0x7f875409f5a0 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8759a1b640 1 librados: init done 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f873c001c80 con 0x7f875409f5a0 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f873c003a70 con 0x7f875409f5a0 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8750ff9640 10 monclient: dump: 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:29.328 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.305+0000 7f8751ffb640 4 set_mon_vals no callback set 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f873c003c10 con 0x7f875409f5a0 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8759a1b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(167..167 src has 1..167) ==== 6193+0+0 (unknown 293837830 0 0) 0x7f873c04ba40 con 0x7f875409f5a0 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8750ff9640 10 client.84125.objecter ms_dispatch 0x7f8754000ba0 osd_map(167..167 src has 1..167) 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8750ff9640 3 client.84125.objecter handle_osd_map got epochs [167,167] > 0 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8750ff9640 3 client.84125.objecter handle_osd_map decoding full epoch 167 2024-09-15T20:35:29.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8759a1b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8759a1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8738005090 con 0x7f875409f5a0 2024-09-15T20:35:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8750ff9640 20 client.84125.objecter dump_active .. 0 homeless 2024-09-15T20:35:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8752ffd640 10 monclient: get_auth_request con 0x7f873403e210 auth_method 0 2024-09-15T20:35:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.309+0000 7f8750ff9640 10 client.84125.objecter ms_handle_connect 0x7f873403e210 2024-09-15T20:35:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.313+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f873c016a10 con 0x7f875409f5a0 2024-09-15T20:35:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.313+0000 7f8750ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:29.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.313+0000 7f8750ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.553+0000 7f8759a1b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:35:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.553+0000 7f8759a1b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:35:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.553+0000 7f8759a1b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:29.561 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.553+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f8738005b90 con 0x7f875409f5a0 2024-09-15T20:35:29.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.557+0000 7f8750ff9640 1 -- 172.21.15.78:0/221872895 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f873c01adf0 con 0x7f875409f5a0 2024-09-15T20:35:29.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.557+0000 7f8750ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:35:29.564 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.557+0000 7f8750ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:29.565 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.557+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.565 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.557+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=1,health=12,kv=11,logm=63} crc {auth=3543251018,config=1747058506,health=3980592879,kv=42821710,logm=96885142}) 2024-09-15T20:35:29.565 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.557+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=11,kv=12,logm=60} crc {auth=4248529570,config=2586702516,health=2176031401,kv=199072906,logm=3630798719}) 2024-09-15T20:35:29.565 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.557+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.566 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.561+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=13,config=1,health=12,kv=11,logm=63} crc {auth=3543251018,config=1747058506,health=3980592879,kv=42821710,logm=96885142}) 2024-09-15T20:35:29.566 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.561+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=12,config=2,health=12,kv=12,logm=62} crc {auth=619829793,config=2586702516,health=3980592879,kv=1152069680,logm=2925796447}) 2024-09-15T20:35:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 10 librados: watch_flush enter 2024-09-15T20:35:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 10 librados: watch_flush exit 2024-09-15T20:35:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 20 client.84125.objecter shutdown clearing up homeless session... 2024-09-15T20:35:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 10 client.84125.objecter successfully canceled tick 2024-09-15T20:35:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 >> v1:172.21.15.146:6812/435598642 conn(0x7f873403e210 legacy=0x7f87340406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 10 monclient: shutdown 2024-09-15T20:35:29.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:29.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 >> v1:172.21.15.78:6790/0 conn(0x7f875409f5a0 legacy=0x7f87541cd8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:29.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 shutdown_connections 2024-09-15T20:35:29.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 >> 172.21.15.78:0/221872895 conn(0x7f87540a63c0 msgr2=0x7f875409dd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:29.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 shutdown_connections 2024-09-15T20:35:29.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 1 -- 172.21.15.78:0/221872895 wait complete. 2024-09-15T20:35:29.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:29.561+0000 7f8759a1b640 1 librados: shutdown 2024-09-15T20:35:29.569 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.561+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.569 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.561+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556176333}) 2024-09-15T20:35:29.569 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.561+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=142039668}) 2024-09-15T20:35:29.569 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.561+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.569 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.561+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1556176333}) 2024-09-15T20:35:29.569 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.561+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=537323310}) 2024-09-15T20:35:29.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116657129}) 2024-09-15T20:35:29.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4228307904}) 2024-09-15T20:35:29.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=116657129}) 2024-09-15T20:35:29.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1836467638}) 2024-09-15T20:35:29.571 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537881160}) 2024-09-15T20:35:29.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1210584579}) 2024-09-15T20:35:29.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2537881160}) 2024-09-15T20:35:29.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.565+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2509027067}) 2024-09-15T20:35:29.573 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.573 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378380108}) 2024-09-15T20:35:29.574 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3133438553}) 2024-09-15T20:35:29.574 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.574 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2378380108}) 2024-09-15T20:35:29.574 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1867117458}) 2024-09-15T20:35:29.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103889875}) 2024-09-15T20:35:29.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245272122}) 2024-09-15T20:35:29.576 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.576 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3103889875}) 2024-09-15T20:35:29.576 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.569+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3093469592}) 2024-09-15T20:35:29.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701247816}) 2024-09-15T20:35:29.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2705999922}) 2024-09-15T20:35:29.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3701247816}) 2024-09-15T20:35:29.578 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1933746036}) 2024-09-15T20:35:29.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67391926}) 2024-09-15T20:35:29.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=719756847}) 2024-09-15T20:35:29.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=67391926}) 2024-09-15T20:35:29.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3154127900}) 2024-09-15T20:35:29.581 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.581 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650728651}) 2024-09-15T20:35:29.581 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.573+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1886698925}) 2024-09-15T20:35:29.581 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.581 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1650728651}) 2024-09-15T20:35:29.581 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=881611876}) 2024-09-15T20:35:29.583 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:35:29.584 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.584 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963953610}) 2024-09-15T20:35:29.584 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4093572443}) 2024-09-15T20:35:29.584 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.584 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963953610}) 2024-09-15T20:35:29.584 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.577+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2582116108}) 2024-09-15T20:35:29.588 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.581+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.591 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.581+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50349707}) 2024-09-15T20:35:29.591 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.581+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2434317890}) 2024-09-15T20:35:29.591 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.581+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.591 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.581+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=50349707}) 2024-09-15T20:35:29.591 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.581+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135649170}) 2024-09-15T20:35:29.596 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.589+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.596 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.589+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=199958954}) 2024-09-15T20:35:29.596 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.589+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2882574869}) 2024-09-15T20:35:29.596 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.589+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.596 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.589+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=199958954}) 2024-09-15T20:35:29.597 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.589+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3307437053}) 2024-09-15T20:35:29.617 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.613+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.618 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.613+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632230011}) 2024-09-15T20:35:29.618 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.613+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2204824677}) 2024-09-15T20:35:29.618 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.613+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.618 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.613+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2632230011}) 2024-09-15T20:35:29.618 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.613+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2819858049}) 2024-09-15T20:35:29.627 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.621+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.627 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.621+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556554543}) 2024-09-15T20:35:29.627 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.621+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=915204450}) 2024-09-15T20:35:29.627 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.621+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.627 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.621+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3556554543}) 2024-09-15T20:35:29.628 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.621+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=849818976}) 2024-09-15T20:35:29.630 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.630 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1184524238}) 2024-09-15T20:35:29.630 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2982507318}) 2024-09-15T20:35:29.631 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.631 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1184524238}) 2024-09-15T20:35:29.631 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3496232415}) 2024-09-15T20:35:29.632 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.632 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530495904}) 2024-09-15T20:35:29.632 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=383980756}) 2024-09-15T20:35:29.632 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.633 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2530495904}) 2024-09-15T20:35:29.633 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.625+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2870744612}) 2024-09-15T20:35:29.634 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.634 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267531933}) 2024-09-15T20:35:29.634 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=593159599}) 2024-09-15T20:35:29.634 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.634 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4267531933}) 2024-09-15T20:35:29.634 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=914379514}) 2024-09-15T20:35:29.635 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.635 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115385616}) 2024-09-15T20:35:29.636 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=440416788}) 2024-09-15T20:35:29.636 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.636 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2115385616}) 2024-09-15T20:35:29.636 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.629+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=541979171}) 2024-09-15T20:35:29.637 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.637 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910491311}) 2024-09-15T20:35:29.637 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1258353247}) 2024-09-15T20:35:29.637 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.637 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1910491311}) 2024-09-15T20:35:29.637 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909116696}) 2024-09-15T20:35:29.638 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.638 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13513701}) 2024-09-15T20:35:29.638 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3094987861}) 2024-09-15T20:35:29.638 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.638 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=13513701}) 2024-09-15T20:35:29.638 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=780017601}) 2024-09-15T20:35:29.640 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.640 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975557300}) 2024-09-15T20:35:29.640 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2047109993}) 2024-09-15T20:35:29.640 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.640 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1975557300}) 2024-09-15T20:35:29.640 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.633+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3764785599}) 2024-09-15T20:35:29.641 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.641 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475589434}) 2024-09-15T20:35:29.641 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3447885928}) 2024-09-15T20:35:29.641 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.641 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3475589434}) 2024-09-15T20:35:29.641 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2458776165}) 2024-09-15T20:35:29.643 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.643 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=397222535,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=100072919,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1681388020,osd_pg_creating=1122281467,osd_snap=2450456513}) 2024-09-15T20:35:29.643 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=40,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=8} crc {logm=4007594420,mdsmap=503983409,mgr=2617012046,mgr_metadata=1997488951,mgrstat=100072919,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=1122281467,osd_snap=611334097}) 2024-09-15T20:35:29.643 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.643 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=36,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=12} crc {logm=397222535,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=100072919,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1681388020,osd_pg_creating=1122281467,osd_snap=2450456513}) 2024-09-15T20:35:29.643 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=38,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=11} crc {logm=2706612501,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=100072919,mon_config_key=3551184854,monmap=3140514009,nvmeofgw=598337686,osd_metadata=4253896995,osd_pg_creating=1122281467,osd_snap=2863297896}) 2024-09-15T20:35:29.644 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.645 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3002046887,osdmap=1678555732}) 2024-09-15T20:35:29.645 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3774908232,osdmap=873155765}) 2024-09-15T20:35:29.645 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.645 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=11,osdmap=89} crc {osd_snap=3002046887,osdmap=1678555732}) 2024-09-15T20:35:29.645 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.637+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=12,osdmap=88} crc {osd_snap=3860074029,osdmap=2676840295}) 2024-09-15T20:35:29.649 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.641+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.649 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.641+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2424632447}) 2024-09-15T20:35:29.649 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.641+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2029263592}) 2024-09-15T20:35:29.649 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.641+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.649 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.641+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2424632447}) 2024-09-15T20:35:29.649 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.641+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2742083944}) 2024-09-15T20:35:29.650 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.645+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.650 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.645+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=69} crc {osdmap=164162428}) 2024-09-15T20:35:29.650 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.645+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=69} crc {osdmap=2916606472}) 2024-09-15T20:35:29.650 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.645+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:35:29.650 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.645+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=69} crc {osdmap=164162428}) 2024-09-15T20:35:29.651 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:35:29.645+0000 7feddf0a3640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=70} crc {osdmap=1778710997}) 2024-09-15T20:35:30.584 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:35:30.585 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:35:30.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient: get_monmap_and_config 2024-09-15T20:35:30.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient: build_initial_monmap 2024-09-15T20:35:30.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient: monmap: 2024-09-15T20:35:30.718 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:30.718 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:30.718 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:30.717999+0000 2024-09-15T20:35:30.718 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:30.717999+0000 2024-09-15T20:35:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:30.719 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:30.719 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:30.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 1 Processor -- start 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 1 -- start start 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient: init 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f12240a0ff0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f1224073800 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f12240a1ae0 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1224072df0 con 0x7f12240a1ae0 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1224074110 con 0x7f1224073800 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12240743f0 con 0x7f12240a0ff0 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:30.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient(hunting): authenticate will time out at 1391.602539s 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1222ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f12240a0ff0 0x7f12240a13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38338/0 (socket says 172.21.15.78:38338) 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1222ffd640 1 -- 172.21.15.78:0/45085586 learned_addr learned my addr 172.21.15.78:0/45085586 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2321393681 0 0) 0x7f12240743f0 con 0x7f12240a0ff0 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient(hunting): my global_id is 84140 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f120c003530 con 0x7f12240a0ff0 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3518039571 0 0) 0x7f1224072df0 con 0x7f12240a1ae0 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:30.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient(hunting): my global_id is 28623 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f12240743f0 con 0x7f12240a1ae0 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1187390032 0 0) 0x7f120c003530 con 0x7f12240a0ff0 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 >> v1:172.21.15.146:6789/0 conn(0x7f1224073800 legacy=0x7f122413ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 >> v1:172.21.15.78:6789/0 conn(0x7f12240a1ae0 legacy=0x7f1224140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 monclient: found mon.noname-b 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1224074eb0 con 0x7f12240a0ff0 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: _finish_auth 0 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:35:30.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f12180016a0 con 0x7f12240a0ff0 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: dump: 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:30.729 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 5 monclient: authenticate success, global_id 84140 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f12180018c0 con 0x7f12240a0ff0 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 1 -- 172.21.15.78:0/45085586 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1218001c90 con 0x7f12240a0ff0 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1221ffb640 10 monclient: dump: 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.730 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.713+0000 7f1229a0e640 4 set_mon_vals no callback set 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: shutdown 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- 172.21.15.78:0/45085586 >> v1:172.21.15.78:6790/0 conn(0x7f12240a0ff0 legacy=0x7f12240a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- 172.21.15.78:0/45085586 shutdown_connections 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- 172.21.15.78:0/45085586 >> 172.21.15.78:0/45085586 conn(0x7f12240a63c0 msgr2=0x7f12240a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- 172.21.15.78:0/45085586 shutdown_connections 2024-09-15T20:35:30.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- 172.21.15.78:0/45085586 wait complete. 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: build_initial_monmap 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: monmap: 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 librados: starting msgr at 2024-09-15T20:35:30.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 librados: starting objecter 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 Processor -- start 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- start start 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 librados: setting wanted keys 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 librados: calling monclient init 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: init 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient(hunting): picked mon.noname-c con 0x7f1224073800 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient(hunting): picked mon.noname-b con 0x7f12240a0ff0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient(hunting): picked mon.noname-a con 0x7f12240a1ae0 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122411fc80 con 0x7f12240a1ae0 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122411d400 con 0x7f12240a0ff0 2024-09-15T20:35:30.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f122409dcd0 con 0x7f1224073800 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient(hunting): authenticate will time out at 1391.605788s 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f12237fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f12240a1ae0 0x7f12241d4f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55524/0 (socket says 172.21.15.78:55524) 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f12237fe640 1 -- 172.21.15.78:0/1960679826 learned_addr learned my addr 172.21.15.78:0/1960679826 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1222ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1224073800 0x7f122411c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38354/0 (socket says 172.21.15.78:38354) 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 client.?.objecter ms_handle_connect 0x7f1224073800 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 client.?.objecter ms_handle_connect 0x7f12240a1ae0 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 client.?.objecter ms_handle_connect 0x7f12240a0ff0 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2878127323 0 0) 0x7f122409dcd0 con 0x7f1224073800 2024-09-15T20:35:30.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): my global_id is 84143 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1210004ec0 con 0x7f1224073800 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4227823600 0 0) 0x7f122411fc80 con 0x7f12240a1ae0 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): my global_id is 28626 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f122409dcd0 con 0x7f12240a1ae0 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 675977695 0 0) 0x7f122411d400 con 0x7f12240a0ff0 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:30.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): my global_id is 23206 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f122411fc80 con 0x7f12240a0ff0 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1400893364 0 0) 0x7f1210004ec0 con 0x7f1224073800 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f122411d400 con 0x7f1224073800 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1218001c10 con 0x7f1224073800 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): dump: 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.736 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1872116368 0 0) 0x7f122409dcd0 con 0x7f12240a1ae0 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1210004ec0 con 0x7f12240a1ae0 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f121c001c40 con 0x7f12240a1ae0 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient(hunting): dump: 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:30.737 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1306621852 0 0) 0x7f122411d400 con 0x7f1224073800 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 >> v1:172.21.15.146:6789/0 conn(0x7f12240a0ff0 legacy=0x7f122411ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 >> v1:172.21.15.78:6789/0 conn(0x7f12240a1ae0 legacy=0x7f12241d4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 monclient: found mon.c 2024-09-15T20:35:30.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f122409d180 con 0x7f1224073800 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient: _finish_auth 0 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 5 monclient: authenticate success, global_id 84143 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: _renew_subs 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f122411ee10 con 0x7f1224073800 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 client.84143.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: _renew_subs 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:30.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1218004c00 con 0x7f1224073800 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1224138650 con 0x7f1224073800 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 librados: init done 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1218004f50 con 0x7f1224073800 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 10 monclient: dump: 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:30.740 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f12217fa640 4 set_mon_vals no callback set 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1218010fc0 con 0x7f1224073800 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.717+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f122411f570 con 0x7f1224073800 2024-09-15T20:35:30.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.721+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(169..169 src has 1..169) ==== 5330+0+0 (unknown 1338376868 0 0) 0x7f121804b790 con 0x7f1224073800 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.721+0000 7f120bfff640 10 client.84143.objecter ms_dispatch 0x7f1224000ba0 osd_map(169..169 src has 1..169) 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.721+0000 7f120bfff640 3 client.84143.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.721+0000 7f120bfff640 3 client.84143.objecter handle_osd_map decoding full epoch 169 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.721+0000 7f120bfff640 20 client.84143.objecter dump_active .. 0 homeless 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.721+0000 7f12227fc640 10 monclient: get_auth_request con 0x7f121003e260 auth_method 0 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.721+0000 7f120bfff640 10 client.84143.objecter ms_handle_connect 0x7f121003e260 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.725+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1218016b60 con 0x7f1224073800 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.725+0000 7f120bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:30.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.725+0000 7f120bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.941+0000 7f1229a0e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:35:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.941+0000 7f1229a0e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:30.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.941+0000 7f1229a0e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.941+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f122413d120 con 0x7f1224073800 2024-09-15T20:35:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.941+0000 7f120bfff640 1 -- 172.21.15.78:0/1960679826 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 899701813) 0x7f1218016460 con 0x7f1224073800 2024-09-15T20:35:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.941+0000 7f120bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:30.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.941+0000 7f120bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:30.946 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:35:30.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 10 librados: watch_flush enter 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 10 librados: watch_flush exit 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 20 client.84143.objecter shutdown clearing up homeless session... 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 10 client.84143.objecter successfully canceled tick 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 >> v1:172.21.15.146:6812/435598642 conn(0x7f121003e260 legacy=0x7f1210040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 10 monclient: shutdown 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 >> v1:172.21.15.78:6790/0 conn(0x7f1224073800 legacy=0x7f122411c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 shutdown_connections 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 >> 172.21.15.78:0/1960679826 conn(0x7f12240a63c0 msgr2=0x7f12241412b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 shutdown_connections 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 1 -- 172.21.15.78:0/1960679826 wait complete. 2024-09-15T20:35:30.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:30.945+0000 7f1229a0e640 1 librados: shutdown 2024-09-15T20:35:30.964 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:35:30.964 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:35:30.964 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:35:30.964 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.069+0000 7fbeca17f640 10 monclient: get_monmap_and_config 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.069+0000 7fbeca17f640 10 monclient: build_initial_monmap 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient: monmap: 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:31.076942+0000 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:31.076942+0000 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.077 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:31.078 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:31.078 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 Processor -- start 2024-09-15T20:35:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- start start 2024-09-15T20:35:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient: init 2024-09-15T20:35:31.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient(hunting): picked mon.noname-c con 0x7fbec40a0620 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient(hunting): picked mon.noname-a con 0x7fbec4134880 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient(hunting): picked mon.noname-b con 0x7fbec40a0a00 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec4073080 con 0x7fbec4134880 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec40a4120 con 0x7fbec40a0620 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec4074300 con 0x7fbec40a0a00 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient(hunting): authenticate will time out at 1391.961724s 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec3fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fbec40a0a00 0x7fbec4140c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38358/0 (socket says 172.21.15.78:38358) 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec2ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbec4134880 0x7fbec40a3a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55534/0 (socket says 172.21.15.78:55534) 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec3fff640 1 -- 172.21.15.78:0/418981284 learned_addr learned my addr 172.21.15.78:0/418981284 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3688038819 0 0) 0x7fbec4073080 con 0x7fbec4134880 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient(hunting): my global_id is 28629 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbeb0003530 con 0x7fbec4134880 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2728144966 0 0) 0x7fbec4074300 con 0x7fbec40a0a00 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient(hunting): my global_id is 84146 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec4073080 con 0x7fbec40a0a00 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1578127292 0 0) 0x7fbeb0003530 con 0x7fbec4134880 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 >> v1:172.21.15.78:6790/0 conn(0x7fbec40a0a00 legacy=0x7fbec4140c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 >> v1:172.21.15.146:6789/0 conn(0x7fbec40a0620 legacy=0x7fbec40a3300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: _finish_hunting 0 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 monclient: found mon.noname-a 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec409cbc0 con 0x7fbec4134880 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: _finish_auth 0 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: _check_auth_tickets 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbeac0016a0 con 0x7fbec4134880 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 5 monclient: authenticate success, global_id 28629 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:35:31.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: dump: 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbeac001840 con 0x7fbec4134880 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:31.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 4 set_mon_vals no callback set 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 1 -- 172.21.15.78:0/418981284 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbeac001c10 con 0x7fbec4134880 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbec27fc640 10 monclient: dump: 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.092 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient: shutdown 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- 172.21.15.78:0/418981284 >> v1:172.21.15.78:6789/0 conn(0x7fbec4134880 legacy=0x7fbec40a3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- 172.21.15.78:0/418981284 shutdown_connections 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- 172.21.15.78:0/418981284 >> 172.21.15.78:0/418981284 conn(0x7fbec40a5790 msgr2=0x7fbec40a5ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- 172.21.15.78:0/418981284 shutdown_connections 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 -- 172.21.15.78:0/418981284 wait complete. 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient: build_initial_monmap 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 10 monclient: monmap: 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:31.093 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 librados: starting msgr at 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.073+0000 7fbeca17f640 1 librados: starting objecter 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 Processor -- start 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 -- start start 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 librados: setting wanted keys 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 librados: calling monclient init 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient: init 2024-09-15T20:35:31.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient(hunting): picked mon.noname-b con 0x7fbec40a0620 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient(hunting): picked mon.noname-c con 0x7fbec40a0a00 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient(hunting): picked mon.noname-a con 0x7fbec4134880 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec4125ae0 con 0x7fbec4134880 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec4125dc0 con 0x7fbec40a0620 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbec4122fc0 con 0x7fbec40a0a00 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient(hunting): authenticate will time out at 1391.964552s 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbec2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fbec40a0a00 0x7fbec41228b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38360/0 (socket says 172.21.15.78:38360) 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbec2ffd640 1 -- 172.21.15.78:0/3247068741 learned_addr learned my addr 172.21.15.78:0/3247068741 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbec3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbec4134880 0x7fbec41d4bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55550/0 (socket says 172.21.15.78:55550) 2024-09-15T20:35:31.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.?.objecter ms_handle_connect 0x7fbec40a0a00 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.?.objecter ms_handle_connect 0x7fbec4134880 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2391553043 0 0) 0x7fbec4122fc0 con 0x7fbec40a0a00 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): my global_id is 84149 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbea4004ec0 con 0x7fbec40a0a00 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988846784 0 0) 0x7fbec4125ae0 con 0x7fbec4134880 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): my global_id is 28632 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec4122fc0 con 0x7fbec4134880 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.?.objecter ms_handle_connect 0x7fbec40a0620 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1610657222 0 0) 0x7fbea4004ec0 con 0x7fbec40a0a00 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbec4125ae0 con 0x7fbec40a0a00 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbeac0015e0 con 0x7fbec40a0a00 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): dump: 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.097 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2019843208 0 0) 0x7fbec4122fc0 con 0x7fbec4134880 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbea4004ec0 con 0x7fbec4134880 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbeb40018c0 con 0x7fbec4134880 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:31.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): dump: 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1972389288 0 0) 0x7fbec4125ae0 con 0x7fbec40a0a00 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 >> v1:172.21.15.146:6789/0 conn(0x7fbec40a0620 legacy=0x7fbec41221a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 >> v1:172.21.15.78:6789/0 conn(0x7fbec4134880 legacy=0x7fbec41d4bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: _finish_hunting 0 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 monclient: found mon.c 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbec40a1920 con 0x7fbec40a0a00 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: _finish_auth 0 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: _check_auth_tickets 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 5 monclient: authenticate success, global_id 84149 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient: _renew_subs 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbec4074cf0 con 0x7fbec40a0a00 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 client.84149.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:31.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient: _renew_subs 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fbec4131d90 con 0x7fbec40a0a00 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 librados: init done 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbeac001800 con 0x7fbec40a0a00 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbeac004c40 con 0x7fbec40a0a00 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbec1ffb640 4 set_mon_vals no callback set 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: dump: 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.101 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fbeac004de0 con 0x7fbec40a0a00 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbea80065d0 con 0x7fbea8002870 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.84149.objecter ms_handle_connect 0x7fbea8002870 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.84149.objecter resend_mon_ops 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(169..169 src has 1..169) ==== 5330+0+0 (unknown 1338376868 0 0) 0x7fbeac04b960 con 0x7fbec40a0a00 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.84149.objecter ms_dispatch 0x7fbec4000ba0 osd_map(169..169 src has 1..169) 2024-09-15T20:35:31.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 3 client.84149.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 3 client.84149.objecter handle_osd_map decoding full epoch 169 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 20 client.84149.objecter dump_active .. 0 homeless 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 351931235 0 0) 0x7fbea80065d0 con 0x7fbea8002870 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient(hunting): my global_id is 28635 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbec4125ae0 con 0x7fbea8002870 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbec3fff640 10 monclient: get_auth_request con 0x7fbea403e040 auth_method 0 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3917076554 0 0) 0x7fbec4125ae0 con 0x7fbea8002870 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbea80065d0 con 0x7fbea8002870 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbebc001900 con 0x7fbea8002870 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: dump: 2024-09-15T20:35:31.103 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 606814142 0 0) 0x7fbea80065d0 con 0x7fbea8002870 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbea800a170 con 0x7fbea8002870 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 client.84149.objecter ms_handle_connect 0x7fbea403e040 2024-09-15T20:35:31.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbebc0049d0 con 0x7fbea8002870 2024-09-15T20:35:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 >> v1:172.21.15.78:6789/0 conn(0x7fbea8002870 legacy=0x7fbea8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.077+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 >> v1:172.21.15.78:6789/0 conn(0x7fbea8002870 legacy=0x7fbea8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:35:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbea8009e80 con 0x7fbea8009140 2024-09-15T20:35:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 client.84149.objecter ms_handle_connect 0x7fbea8009140 2024-09-15T20:35:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 client.84149.objecter resend_mon_ops 2024-09-15T20:35:31.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1728995960 0 0) 0x7fbea8009e80 con 0x7fbea8009140 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient(hunting): my global_id is 28638 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbea80065d0 con 0x7fbea8009140 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2624307400 0 0) 0x7fbea80065d0 con 0x7fbea8009140 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbea8009e80 con 0x7fbea8009140 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbebc002030 con 0x7fbea8009140 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient: dump: 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.107 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2161360271 0 0) 0x7fbea8009e80 con 0x7fbea8009140 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbea800c420 con 0x7fbea8009140 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4140 (unknown 0 0 1235068986) 0x7fbebc0020b0 con 0x7fbea8009140 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 >> v1:172.21.15.78:6789/0 conn(0x7fbea8009140 legacy=0x7fbea80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbebbfff640 1 -- 172.21.15.78:0/3247068741 >> v1:172.21.15.78:6789/0 conn(0x7fbea8009140 legacy=0x7fbea80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 16, 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 243636, 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:35:31.109 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.110 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.111 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:35:31.112 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.113 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:35:31.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:35:31.115 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.116 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:31.117 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.118 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:35:31.119 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:35:31.120 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:35:31.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 10 librados: watch_flush enter 2024-09-15T20:35:31.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 10 librados: watch_flush exit 2024-09-15T20:35:31.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 20 client.84149.objecter shutdown clearing up homeless session... 2024-09-15T20:35:31.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 10 client.84149.objecter successfully canceled tick 2024-09-15T20:35:31.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 >> v1:172.21.15.146:6812/435598642 conn(0x7fbea403e040 legacy=0x7fbea4040500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 10 monclient: shutdown 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 >> v1:172.21.15.78:6790/0 conn(0x7fbec40a0a00 legacy=0x7fbec41228b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbec3fff640 1 -- 172.21.15.78:0/3247068741 reap_dead start 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbec3fff640 1 -- 172.21.15.78:0/3247068741 reap_dead start 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 shutdown_connections 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 >> 172.21.15.78:0/3247068741 conn(0x7fbec40a5790 msgr2=0x7fbec40a3170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 shutdown_connections 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 1 -- 172.21.15.78:0/3247068741 wait complete. 2024-09-15T20:35:31.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.085+0000 7fbeca17f640 1 librados: shutdown 2024-09-15T20:35:31.123 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:35:31.124 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:35:31.124 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:35:31.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient: get_monmap_and_config 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient: build_initial_monmap 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient: monmap: 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:31.237536+0000 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:31.237536+0000 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:31.238 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:31.239 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 1 Processor -- start 2024-09-15T20:35:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 1 -- start start 2024-09-15T20:35:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient: init 2024-09-15T20:35:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:35:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient(hunting): picked mon.noname-b con 0x7f3f340a0ff0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:31.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient(hunting): picked mon.noname-a con 0x7f3f34073800 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient(hunting): picked mon.noname-c con 0x7f3f340a1ae0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f34072df0 con 0x7f3f34073800 2024-09-15T20:35:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f34074110 con 0x7f3f340a1ae0 2024-09-15T20:35:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f340743f0 con 0x7f3f340a0ff0 2024-09-15T20:35:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient(hunting): authenticate will time out at 1392.122626s 2024-09-15T20:35:31.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f32ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f3f340a0ff0 0x7f3f340a13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38366/0 (socket says 172.21.15.78:38366) 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f32ffd640 1 -- 172.21.15.78:0/1865659319 learned_addr learned my addr 172.21.15.78:0/1865659319 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2832877254 0 0) 0x7f3f340743f0 con 0x7f3f340a0ff0 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient(hunting): my global_id is 84152 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f20003530 con 0x7f3f340a0ff0 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1508276473 0 0) 0x7f3f34072df0 con 0x7f3f34073800 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient(hunting): my global_id is 28641 2024-09-15T20:35:31.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f340743f0 con 0x7f3f34073800 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 660289711 0 0) 0x7f3f340743f0 con 0x7f3f34073800 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 >> v1:172.21.15.78:6790/0 conn(0x7f3f340a0ff0 legacy=0x7f3f340a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 >> v1:172.21.15.146:6789/0 conn(0x7f3f340a1ae0 legacy=0x7f3f34140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: _finish_hunting 0 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 monclient: found mon.noname-a 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f34074eb0 con 0x7f3f34073800 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: _finish_auth 0 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: _check_auth_tickets 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f1c001680 con 0x7f3f34073800 2024-09-15T20:35:31.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 5 monclient: authenticate success, global_id 28641 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: dump: 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.245 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f1c0018a0 con 0x7f3f34073800 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 1 -- 172.21.15.78:0/1865659319 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f1c001cb0 con 0x7f3f34073800 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f39847640 4 set_mon_vals no callback set 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.233+0000 7f3f327fc640 10 monclient: dump: 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.246 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: shutdown 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- 172.21.15.78:0/1865659319 >> v1:172.21.15.78:6789/0 conn(0x7f3f34073800 legacy=0x7f3f3413ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- 172.21.15.78:0/1865659319 shutdown_connections 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- 172.21.15.78:0/1865659319 >> 172.21.15.78:0/1865659319 conn(0x7f3f340a63c0 msgr2=0x7f3f340a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- 172.21.15.78:0/1865659319 shutdown_connections 2024-09-15T20:35:31.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- 172.21.15.78:0/1865659319 wait complete. 2024-09-15T20:35:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: build_initial_monmap 2024-09-15T20:35:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: monmap: 2024-09-15T20:35:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:31.253 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 librados: starting msgr at 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 librados: starting objecter 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 Processor -- start 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- start start 2024-09-15T20:35:31.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 librados: setting wanted keys 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 librados: calling monclient init 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: init 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient(hunting): picked mon.noname-a con 0x7f3f34073800 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient(hunting): picked mon.noname-c con 0x7f3f340a0ff0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient(hunting): picked mon.noname-b con 0x7f3f340a1ae0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f3409dba0 con 0x7f3f34073800 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f3409d180 con 0x7f3f340a1ae0 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f3413a560 con 0x7f3f340a0ff0 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:31.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient(hunting): authenticate will time out at 1392.125627s 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f2a7fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f3f340a0ff0 0x7f3f341376d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38380/0 (socket says 172.21.15.78:38380) 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f32ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3f34073800 0x7f3f34136fc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55580/0 (socket says 172.21.15.78:55580) 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f2a7fc640 1 -- 172.21.15.78:0/1084654148 learned_addr learned my addr 172.21.15.78:0/1084654148 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.?.objecter ms_handle_connect 0x7f3f340a0ff0 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.?.objecter ms_handle_connect 0x7f3f34073800 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3495761592 0 0) 0x7f3f3413a560 con 0x7f3f340a0ff0 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): my global_id is 84155 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f14004ec0 con 0x7f3f340a0ff0 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204628470 0 0) 0x7f3f3409dba0 con 0x7f3f34073800 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): my global_id is 28644 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f3413a560 con 0x7f3f34073800 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.?.objecter ms_handle_connect 0x7f3f340a1ae0 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1071938783 0 0) 0x7f3f14004ec0 con 0x7f3f340a0ff0 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f3409dba0 con 0x7f3f340a0ff0 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f1c001c30 con 0x7f3f340a0ff0 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:31.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): dump: 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1579756368 0 0) 0x7f3f3413a560 con 0x7f3f34073800 2024-09-15T20:35:31.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f14004ec0 con 0x7f3f34073800 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f240012e0 con 0x7f3f34073800 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): dump: 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.259 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4002833966 0 0) 0x7f3f3409dba0 con 0x7f3f340a0ff0 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 >> v1:172.21.15.146:6789/0 conn(0x7f3f340a1ae0 legacy=0x7f3f34137de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 >> v1:172.21.15.78:6789/0 conn(0x7f3f34073800 legacy=0x7f3f34136fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 monclient: found mon.c 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f3413b220 con 0x7f3f340a0ff0 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient: _finish_auth 0 2024-09-15T20:35:31.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 5 monclient: authenticate success, global_id 84155 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: _renew_subs 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f34139600 con 0x7f3f340a0ff0 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 client.84155.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: _renew_subs 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f341d6160 con 0x7f3f340a0ff0 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 librados: init done 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f1c001800 con 0x7f3f340a0ff0 2024-09-15T20:35:31.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f1c003b10 con 0x7f3f340a0ff0 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f317fa640 4 set_mon_vals no callback set 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient: dump: 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.262 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3f180065d0 con 0x7f3f18002870 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.84155.objecter ms_handle_connect 0x7f3f18002870 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.84155.objecter resend_mon_ops 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f3f1c003cb0 con 0x7f3f340a0ff0 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(169..169 src has 1..169) ==== 5330+0+0 (unknown 1338376868 0 0) 0x7f3f1c04b770 con 0x7f3f340a0ff0 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 client.84155.objecter ms_dispatch 0x7f3f34000ba0 osd_map(169..169 src has 1..169) 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 3 client.84155.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 3 client.84155.objecter handle_osd_map decoding full epoch 169 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 20 client.84155.objecter dump_active .. 0 homeless 2024-09-15T20:35:31.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 932761177 0 0) 0x7f3f180065d0 con 0x7f3f18002870 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 10 monclient(hunting): my global_id is 84158 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f3409dba0 con 0x7f3f18002870 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 204868555 0 0) 0x7f3f3409dba0 con 0x7f3f18002870 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f180065d0 con 0x7f3f18002870 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f240047f0 con 0x7f3f18002870 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.237+0000 7f3f337fe640 10 monclient: get_auth_request con 0x7f3f1403e0d0 auth_method 0 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:31.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 10 monclient: dump: 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3799993303 0 0) 0x7f3f180065d0 con 0x7f3f18002870 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3f1800a170 con 0x7f3f18002870 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 10 client.84155.objecter ms_handle_connect 0x7f3f1403e0d0 2024-09-15T20:35:31.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f3f240047f0 con 0x7f3f18002870 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 >> v1:172.21.15.78:6790/0 conn(0x7f3f18002870 legacy=0x7f3f18004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.241+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 >> v1:172.21.15.78:6790/0 conn(0x7f3f18002870 legacy=0x7f3f18004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3f18009e80 con 0x7f3f18009140 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 client.84155.objecter ms_handle_connect 0x7f3f18009140 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 client.84155.objecter resend_mon_ops 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1623120844 0 0) 0x7f3f18009e80 con 0x7f3f18009140 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient(hunting): my global_id is 84161 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f180065d0 con 0x7f3f18009140 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2790452102 0 0) 0x7f3f180065d0 con 0x7f3f18009140 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f18009e80 con 0x7f3f18009140 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f24004fb0 con 0x7f3f18009140 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient: dump: 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.267 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3080674128 0 0) 0x7f3f18009e80 con 0x7f3f18009140 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3f1800c400 con 0x7f3f18009140 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3969 (unknown 0 0 347341864) 0x7f3f24006540 con 0x7f3f18009140 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 >> v1:172.21.15.78:6790/0 conn(0x7f3f18009140 legacy=0x7f3f180095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f30ff9640 1 -- 172.21.15.78:0/1084654148 >> v1:172.21.15.78:6790/0 conn(0x7f3f18009140 legacy=0x7f3f180095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 16, 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 2931, 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:35:31.269 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.271 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:35:31.272 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.273 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.274 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.276 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:31.277 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.278 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:35:31.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:35:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.280 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:35:31.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 10 librados: watch_flush enter 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 10 librados: watch_flush exit 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 20 client.84155.objecter shutdown clearing up homeless session... 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 10 client.84155.objecter successfully canceled tick 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 >> v1:172.21.15.146:6812/435598642 conn(0x7f3f1403e0d0 legacy=0x7f3f14040590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 10 monclient: shutdown 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 >> v1:172.21.15.78:6790/0 conn(0x7f3f340a0ff0 legacy=0x7f3f341376d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f337fe640 1 -- 172.21.15.78:0/1084654148 reap_dead start 2024-09-15T20:35:31.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f337fe640 1 -- 172.21.15.78:0/1084654148 reap_dead start 2024-09-15T20:35:31.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 shutdown_connections 2024-09-15T20:35:31.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 >> 172.21.15.78:0/1084654148 conn(0x7f3f340a63c0 msgr2=0x7f3f34130ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:31.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 shutdown_connections 2024-09-15T20:35:31.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 1 -- 172.21.15.78:0/1084654148 wait complete. 2024-09-15T20:35:31.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.245+0000 7f3f39847640 1 librados: shutdown 2024-09-15T20:35:31.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:35:31.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:35:31.284 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:35:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 10 monclient: get_monmap_and_config 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 10 monclient: build_initial_monmap 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 10 monclient: monmap: 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:31.399499+0000 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:31.399499+0000 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:31.401 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 1 Processor -- start 2024-09-15T20:35:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 1 -- start start 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 10 monclient: init 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 10 monclient(hunting): picked mon.noname-c con 0x7f0234135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.393+0000 7f023bf9f640 10 monclient(hunting): picked mon.noname-a con 0x7f02340a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient(hunting): picked mon.noname-b con 0x7f023413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f02340745e0 con 0x7f02340a24e0 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023406ecf0 con 0x7f0234135b60 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023406efd0 con 0x7f023413c220 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient(hunting): authenticate will time out at 1392.284054s 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023953a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f02340a24e0 0x7f023413ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55586/0 (socket says 172.21.15.78:55586) 2024-09-15T20:35:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023a53c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f023413c220 0x7f023413e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38424/0 (socket says 172.21.15.78:38424) 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023953a640 1 -- 172.21.15.78:0/1070347151 learned_addr learned my addr 172.21.15.78:0/1070347151 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2893195816 0 0) 0x7f023406efd0 con 0x7f023413c220 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient(hunting): my global_id is 84164 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0220003530 con 0x7f023413c220 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1329655353 0 0) 0x7f02340745e0 con 0x7f02340a24e0 2024-09-15T20:35:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient(hunting): my global_id is 28647 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f023406efd0 con 0x7f02340a24e0 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1396326458 0 0) 0x7f0220003530 con 0x7f023413c220 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 >> v1:172.21.15.146:6789/0 conn(0x7f0234135b60 legacy=0x7f0234135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 >> v1:172.21.15.78:6789/0 conn(0x7f02340a24e0 legacy=0x7f023413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: _finish_hunting 0 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 monclient: found mon.noname-b 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0234136650 con 0x7f023413c220 2024-09-15T20:35:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: _finish_auth 0 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: _check_auth_tickets 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f022c001a20 con 0x7f023413c220 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 5 monclient: authenticate success, global_id 84164 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: dump: 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f022c001c40 con 0x7f023413c220 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 1 -- 172.21.15.78:0/1070347151 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f022c002050 con 0x7f023413c220 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 4 set_mon_vals no callback set 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f0238d39640 10 monclient: dump: 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: shutdown 2024-09-15T20:35:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- 172.21.15.78:0/1070347151 >> v1:172.21.15.78:6790/0 conn(0x7f023413c220 legacy=0x7f023413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- 172.21.15.78:0/1070347151 shutdown_connections 2024-09-15T20:35:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- 172.21.15.78:0/1070347151 >> 172.21.15.78:0/1070347151 conn(0x7f02340a63c0 msgr2=0x7f02340a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- 172.21.15.78:0/1070347151 shutdown_connections 2024-09-15T20:35:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- 172.21.15.78:0/1070347151 wait complete. 2024-09-15T20:35:31.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: build_initial_monmap 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: monmap: 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 librados: starting msgr at 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 librados: starting objecter 2024-09-15T20:35:31.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 Processor -- start 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- start start 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 librados: setting wanted keys 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 librados: calling monclient init 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: init 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient(hunting): picked mon.noname-b con 0x7f02340a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient(hunting): picked mon.noname-c con 0x7f0234135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient(hunting): picked mon.noname-a con 0x7f023413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023413bc40 con 0x7f023413c220 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023413fb00 con 0x7f02340a24e0 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f023409d180 con 0x7f0234135b60 2024-09-15T20:35:31.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient(hunting): authenticate will time out at 1392.287097s 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023953a640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f0234135b60 0x7f0234072300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38438/0 (socket says 172.21.15.78:38438) 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023a53c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f023413c220 0x7f02341d4c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55598/0 (socket says 172.21.15.78:55598) 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023953a640 1 -- 172.21.15.78:0/2868689536 learned_addr learned my addr 172.21.15.78:0/2868689536 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 client.?.objecter ms_handle_connect 0x7f0234135b60 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 client.?.objecter ms_handle_connect 0x7f023413c220 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1120641982 0 0) 0x7f023409d180 con 0x7f0234135b60 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): my global_id is 84167 2024-09-15T20:35:31.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0218004ec0 con 0x7f0234135b60 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2578150695 0 0) 0x7f023413bc40 con 0x7f023413c220 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): my global_id is 28650 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f023409d180 con 0x7f023413c220 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 client.?.objecter ms_handle_connect 0x7f02340a24e0 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2968992074 0 0) 0x7f0218004ec0 con 0x7f0234135b60 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f023413bc40 con 0x7f0234135b60 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f02240012e0 con 0x7f0234135b60 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): dump: 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.421 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 593023852 0 0) 0x7f023409d180 con 0x7f023413c220 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0218004ec0 con 0x7f023413c220 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f022c001df0 con 0x7f023413c220 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): dump: 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.422 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2794086912 0 0) 0x7f023413fb00 con 0x7f02340a24e0 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient(hunting): my global_id is 23224 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f023409d180 con 0x7f02340a24e0 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3565770544 0 0) 0x7f023413bc40 con 0x7f0234135b60 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 >> v1:172.21.15.146:6789/0 conn(0x7f02340a24e0 legacy=0x7f0234071bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 >> v1:172.21.15.78:6789/0 conn(0x7f023413c220 legacy=0x7f02341d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 monclient: found mon.c 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f023409e140 con 0x7f0234135b60 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient: _finish_auth 0 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f022a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 5 monclient: authenticate success, global_id 84167 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: _renew_subs 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.397+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f023409e420 con 0x7f0234135b60 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f023bf9f640 10 client.84167.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f023bf9f640 10 monclient: _renew_subs 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f023bf9f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f023413af70 con 0x7f0234135b60 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f023bf9f640 1 librados: init done 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f02240047f0 con 0x7f0234135b60 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0224004b40 con 0x7f0234135b60 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: dump: 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.425 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022bfff640 4 set_mon_vals no callback set 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f023bf9f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f021c0065d0 con 0x7f021c002870 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f0224010bb0 con 0x7f0234135b60 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(169..169 src has 1..169) ==== 5330+0+0 (unknown 1338376868 0 0) 0x7f022404b7f0 con 0x7f0234135b60 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 client.84167.objecter ms_dispatch 0x7f0234000ba0 osd_map(169..169 src has 1..169) 2024-09-15T20:35:31.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 3 client.84167.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 3 client.84167.objecter handle_osd_map decoding full epoch 169 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 20 client.84167.objecter dump_active .. 0 homeless 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 client.84167.objecter ms_handle_connect 0x7f021c002870 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 client.84167.objecter resend_mon_ops 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f023a53c640 10 monclient: get_auth_request con 0x7f021803e240 auth_method 0 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2168546110 0 0) 0x7f021c0065d0 con 0x7f021c002870 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient(hunting): my global_id is 23227 2024-09-15T20:35:31.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f023413bc40 con 0x7f021c002870 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 client.84167.objecter ms_handle_connect 0x7f021803e240 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2056688281 0 0) 0x7f023413bc40 con 0x7f021c002870 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f021c0065d0 con 0x7f021c002870 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f02300017e0 con 0x7f021c002870 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: dump: 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.428 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1124852824 0 0) 0x7f021c0065d0 con 0x7f021c002870 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f021c00a170 con 0x7f021c002870 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f0230001b00 con 0x7f021c002870 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:31.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 >> v1:172.21.15.146:6789/0 conn(0x7f021c002870 legacy=0x7f021c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.401+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 >> v1:172.21.15.146:6789/0 conn(0x7f021c002870 legacy=0x7f021c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f023bf9f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:35:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f021c009e80 con 0x7f021c009140 2024-09-15T20:35:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 client.84167.objecter ms_handle_connect 0x7f021c009140 2024-09-15T20:35:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 client.84167.objecter resend_mon_ops 2024-09-15T20:35:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 744244469 0 0) 0x7f021c009e80 con 0x7f021c009140 2024-09-15T20:35:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient(hunting): my global_id is 23230 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f021c0065d0 con 0x7f021c009140 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2266779465 0 0) 0x7f021c0065d0 con 0x7f021c009140 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f021c009e80 con 0x7f021c009140 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f02300037a0 con 0x7f021c009140 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient: dump: 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.432 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 642160002 0 0) 0x7f021c009e80 con 0x7f021c009140 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f021c00c420 con 0x7f021c009140 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 2157704054) 0x7f0230003730 con 0x7f021c009140 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 >> v1:172.21.15.146:6789/0 conn(0x7f021c009140 legacy=0x7f021c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.409+0000 7f022a7fc640 1 -- 172.21.15.78:0/2868689536 >> v1:172.21.15.146:6789/0 conn(0x7f021c009140 legacy=0x7f021c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 16, 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 243925, 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:35:31.434 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 3, 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.435 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.436 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:35:31.437 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:31.438 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:35:31.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.442 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.443 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:35:31.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.445 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 8 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:31.446 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:31.447 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:35:31.447 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:35:31.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 10 librados: watch_flush enter 2024-09-15T20:35:31.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 10 librados: watch_flush exit 2024-09-15T20:35:31.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 20 client.84167.objecter shutdown clearing up homeless session... 2024-09-15T20:35:31.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 10 client.84167.objecter successfully canceled tick 2024-09-15T20:35:31.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 >> v1:172.21.15.146:6812/435598642 conn(0x7f021803e240 legacy=0x7f0218040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 10 monclient: shutdown 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 >> v1:172.21.15.78:6790/0 conn(0x7f0234135b60 legacy=0x7f0234072300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023a53c640 1 -- 172.21.15.78:0/2868689536 reap_dead start 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023a53c640 1 -- 172.21.15.78:0/2868689536 reap_dead start 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 shutdown_connections 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 >> 172.21.15.78:0/2868689536 conn(0x7f02340a63c0 msgr2=0x7f023413ed00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 shutdown_connections 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 1 -- 172.21.15.78:0/2868689536 wait complete. 2024-09-15T20:35:31.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.413+0000 7f023bf9f640 1 librados: shutdown 2024-09-15T20:35:31.450 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:35:31.450 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T20:35:31.450 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T20:35:31.450 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T20:35:31.450 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T20:35:31.450 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T20:35:31.450 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:35:31.463 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T20:35:31.463 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:35:31.463 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:35:31.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient: get_monmap_and_config 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient: build_initial_monmap 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient: monmap: 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:31.614794+0000 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:31.614794+0000 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:31.615 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 1 Processor -- start 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 1 -- start start 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient: init 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f0137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f0074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f013e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:31.623 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f0138850 con 0x7fe7f013e370 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f00a5da0 con 0x7fe7f0074500 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f0073560 con 0x7fe7f0137d60 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f6b2e640 10 monclient(hunting): authenticate will time out at 1392.499879s 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.609+0000 7fe7f50cb640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fe7f013e370 legacy=0x7fe7f0140760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f48ca640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe7f0137d60 0x7fe7f0138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38466/0 (socket says 172.21.15.78:38466) 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f48ca640 1 -- 172.21.15.78:0/629278019 learned_addr learned my addr 172.21.15.78:0/629278019 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 205228773 0 0) 0x7fe7f0073560 con 0x7fe7f0137d60 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient(hunting): my global_id is 84170 2024-09-15T20:35:31.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7d8003530 con 0x7fe7f0137d60 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4178646303 0 0) 0x7fe7d8003530 con 0x7fe7f0137d60 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 >> v1:172.21.15.146:6789/0 conn(0x7fe7f0074500 legacy=0x7fe7f013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 >> v1:172.21.15.78:6789/0 conn(0x7fe7f013e370 legacy=0x7fe7f0140760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 monclient: found mon.noname-b 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f0073840 con 0x7fe7f0137d60 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: _finish_auth 0 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe7ec0016a0 con 0x7fe7f0137d60 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 5 monclient: authenticate success, global_id 84170 2024-09-15T20:35:31.625 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: dump: 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.626 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe7ec0018c0 con 0x7fe7f0137d60 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 1 -- 172.21.15.78:0/629278019 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe7ec001cd0 con 0x7fe7f0137d60 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 4 set_mon_vals no callback set 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e77fe640 10 monclient: dump: 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.627 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: shutdown 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/629278019 >> v1:172.21.15.78:6790/0 conn(0x7fe7f0137d60 legacy=0x7fe7f0138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/629278019 shutdown_connections 2024-09-15T20:35:31.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/629278019 >> 172.21.15.78:0/629278019 conn(0x7fe7f00aa650 msgr2=0x7fe7f00aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/629278019 shutdown_connections 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/629278019 wait complete. 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: build_initial_monmap 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: monmap: 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.629 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 librados: starting msgr at 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 librados: starting objecter 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 Processor -- start 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- start start 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 librados: setting wanted keys 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 librados: calling monclient init 2024-09-15T20:35:31.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: init 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient(hunting): picked mon.noname-c con 0x7fe7f0074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient(hunting): picked mon.noname-b con 0x7fe7f0137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient(hunting): picked mon.noname-a con 0x7fe7f013e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f009e0b0 con 0x7fe7f013e370 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f00a0130 con 0x7fe7f0137d60 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7f00a0600 con 0x7fe7f0074500 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f50cb640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fe7f013e370 legacy=0x7fe7f01d4e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f48ca640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe7f0074500 0x7fe7f009f310 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38472/0 (socket says 172.21.15.78:38472) 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f48ca640 1 -- 172.21.15.78:0/3834410658 learned_addr learned my addr 172.21.15.78:0/3834410658 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient(hunting): authenticate will time out at 1392.503087s 2024-09-15T20:35:31.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 client.?.objecter ms_handle_connect 0x7fe7f0074500 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1891293975 0 0) 0x7fe7f00a0600 con 0x7fe7f0074500 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): my global_id is 84173 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7dc004ec0 con 0x7fe7f0074500 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 client.?.objecter ms_handle_connect 0x7fe7f0137d60 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2200697658 0 0) 0x7fe7dc004ec0 con 0x7fe7f0074500 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe7f00a0600 con 0x7fe7f0074500 2024-09-15T20:35:31.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe7ec001c10 con 0x7fe7f0074500 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): dump: 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.633 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1056583455 0 0) 0x7fe7f00a0130 con 0x7fe7f0137d60 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient(hunting): my global_id is 23236 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe7dc004ec0 con 0x7fe7f0137d60 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 631756378 0 0) 0x7fe7f00a0600 con 0x7fe7f0074500 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 >> v1:172.21.15.146:6789/0 conn(0x7fe7f0137d60 legacy=0x7fe7f009fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 >> v1:172.21.15.78:6789/0 conn(0x7fe7f013e370 legacy=0x7fe7f01d4e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient: _finish_hunting 0 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 monclient: found mon.c 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7f0132630 con 0x7fe7f0074500 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient: _finish_auth 0 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient: _check_auth_tickets 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 5 monclient: authenticate success, global_id 84173 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: _renew_subs 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe7f0132910 con 0x7fe7f0074500 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 client.84173.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: _renew_subs 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe7f0140d10 con 0x7fe7f0074500 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7f6b2e640 1 librados: init done 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe7ec004c00 con 0x7fe7f0074500 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.613+0000 7fe7e57fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:31.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe7ec005050 con 0x7fe7f0074500 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e6ffd640 4 set_mon_vals no callback set 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 10 monclient: dump: 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:31.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7f6b2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7f6b2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7f6b2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7d0005090 con 0x7fe7f0074500 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fe7ec0051f0 con 0x7fe7f0074500 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(169..169 src has 1..169) ==== 5330+0+0 (unknown 1338376868 0 0) 0x7fe7ec04b870 con 0x7fe7f0074500 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 10 client.84173.objecter ms_dispatch 0x7fe7f0000ba0 osd_map(169..169 src has 1..169) 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 3 client.84173.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 3 client.84173.objecter handle_osd_map decoding full epoch 169 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 20 client.84173.objecter dump_active .. 0 homeless 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e7fff640 10 monclient: get_auth_request con 0x7fe7dc03e0f0 auth_method 0 2024-09-15T20:35:31.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.617+0000 7fe7e57fa640 10 client.84173.objecter ms_handle_connect 0x7fe7dc03e0f0 2024-09-15T20:35:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.621+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fe7ec01aea0 con 0x7fe7f0074500 2024-09-15T20:35:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.621+0000 7fe7e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:31.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.621+0000 7fe7e57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.837+0000 7fe7f6b2e640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:35:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.837+0000 7fe7f6b2e640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.837+0000 7fe7f6b2e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:31.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.837+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe7d0005380 con 0x7fe7f0074500 2024-09-15T20:35:31.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.837+0000 7fe7e57fa640 1 -- 172.21.15.78:0/3834410658 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3363991095) 0x7fe7ec01fde0 con 0x7fe7f0074500 2024-09-15T20:35:31.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:31.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7e57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:31.845 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 10 librados: watch_flush enter 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 10 librados: watch_flush exit 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 20 client.84173.objecter shutdown clearing up homeless session... 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 10 client.84173.objecter successfully canceled tick 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 >> v1:172.21.15.146:6812/435598642 conn(0x7fe7dc03e0f0 legacy=0x7fe7dc0405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 10 monclient: shutdown 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 >> v1:172.21.15.78:6790/0 conn(0x7fe7f0074500 legacy=0x7fe7f009f310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:31.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 shutdown_connections 2024-09-15T20:35:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 >> 172.21.15.78:0/3834410658 conn(0x7fe7f00aa650 msgr2=0x7fe7f00a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 shutdown_connections 2024-09-15T20:35:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 1 -- 172.21.15.78:0/3834410658 wait complete. 2024-09-15T20:35:31.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:31.841+0000 7fe7f6b2e640 1 librados: shutdown 2024-09-15T20:35:34.862 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:35:34.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.985+0000 7ff129f58640 10 monclient: get_monmap_and_config 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.985+0000 7ff129f58640 10 monclient: build_initial_monmap 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.985+0000 7ff129f58640 10 monclient: monmap: 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:34.991433+0000 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:34.991433+0000 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:34.992 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.985+0000 7ff129f58640 1 Processor -- start 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.985+0000 7ff129f58640 1 -- start start 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.985+0000 7ff129f58640 10 monclient: init 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.985+0000 7ff129f58640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.985+0000 7ff129f58640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): picked mon.noname-c con 0x7ff124135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): picked mon.noname-b con 0x7ff1240a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): picked mon.noname-a con 0x7ff12413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1240745e0 con 0x7ff12413c220 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12406ecf0 con 0x7ff124135b60 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12406efd0 con 0x7ff1240a24e0 2024-09-15T20:35:34.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): authenticate will time out at 1395.876106s 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff123fff640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7ff12413c220 legacy=0x7ff12413e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff122ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff1240a24e0 0x7ff12413ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38476/0 (socket says 172.21.15.78:38476) 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff122ffd640 1 -- 172.21.15.78:0/995812721 learned_addr learned my addr 172.21.15.78:0/995812721 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 75468133 0 0) 0x7ff12406efd0 con 0x7ff1240a24e0 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient(hunting): _negotiate 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient(hunting): my global_id is 84176 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff104003530 con 0x7ff1240a24e0 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2097844341 0 0) 0x7ff104003530 con 0x7ff1240a24e0 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 >> v1:172.21.15.146:6789/0 conn(0x7ff124135b60 legacy=0x7ff124135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:35.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 >> v1:172.21.15.78:6789/0 conn(0x7ff12413c220 legacy=0x7ff12413e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: _finish_hunting 0 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 monclient: found mon.noname-b 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff124136650 con 0x7ff1240a24e0 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: _finish_auth 0 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: _check_auth_tickets 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff10c0016a0 con 0x7ff1240a24e0 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 5 monclient: authenticate success, global_id 84176 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:35.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: dump: 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:35.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff10c0018c0 con 0x7ff1240a24e0 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 1 -- 172.21.15.78:0/995812721 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff10c001cd0 con 0x7ff1240a24e0 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1227fc640 10 monclient: dump: 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:35.003 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 4 set_mon_vals no callback set 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: shutdown 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- 172.21.15.78:0/995812721 >> v1:172.21.15.78:6790/0 conn(0x7ff1240a24e0 legacy=0x7ff12413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- 172.21.15.78:0/995812721 shutdown_connections 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- 172.21.15.78:0/995812721 >> 172.21.15.78:0/995812721 conn(0x7ff1240a63c0 msgr2=0x7ff1240a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- 172.21.15.78:0/995812721 shutdown_connections 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- 172.21.15.78:0/995812721 wait complete. 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: build_initial_monmap 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: monmap: 2024-09-15T20:35:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 librados: starting msgr at 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 librados: starting objecter 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 Processor -- start 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- start start 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 librados: setting wanted keys 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 librados: calling monclient init 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: init 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): picked mon.noname-a con 0x7ff1240a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): picked mon.noname-b con 0x7ff124135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): picked mon.noname-c con 0x7ff12413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff124136a60 con 0x7ff1240a24e0 2024-09-15T20:35:35.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff124138390 con 0x7ff124135b60 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff124135060 con 0x7ff12413c220 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient(hunting): authenticate will time out at 1395.879041s 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff1237fe640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7ff1240a24e0 legacy=0x7ff124071d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff123fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff12413c220 0x7ff1241d2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38492/0 (socket says 172.21.15.78:38492) 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff123fff640 1 -- 172.21.15.78:0/2403229891 learned_addr learned my addr 172.21.15.78:0/2403229891 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 client.?.objecter ms_handle_connect 0x7ff12413c220 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3515015215 0 0) 0x7ff124135060 con 0x7ff12413c220 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:35.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): my global_id is 84179 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff108004ec0 con 0x7ff12413c220 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 client.?.objecter ms_handle_connect 0x7ff124135b60 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3817214409 0 0) 0x7ff108004ec0 con 0x7ff12413c220 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff124135060 con 0x7ff12413c220 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1140040c0 con 0x7ff12413c220 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): dump: 2024-09-15T20:35:35.008 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2531928997 0 0) 0x7ff124138390 con 0x7ff124135b60 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:35.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:35.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient(hunting): my global_id is 23242 2024-09-15T20:35:35.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff108004ec0 con 0x7ff124135b60 2024-09-15T20:35:35.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1673892889 0 0) 0x7ff124135060 con 0x7ff12413c220 2024-09-15T20:35:35.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 >> v1:172.21.15.146:6789/0 conn(0x7ff124135b60 legacy=0x7ff124072470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:35.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 >> v1:172.21.15.78:6789/0 conn(0x7ff1240a24e0 legacy=0x7ff124071d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:35:35.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:35:35.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 monclient: found mon.c 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12413bbb0 con 0x7ff12413c220 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient: _finish_auth 0 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 5 monclient: authenticate success, global_id 84179 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: _renew_subs 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1240a3760 con 0x7ff12413c220 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 client.84179.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: _renew_subs 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff124136e70 con 0x7ff12413c220 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff129f58640 1 librados: init done 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff114004260 con 0x7ff12413c220 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff121ffb640 4 set_mon_vals no callback set 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.989+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1140046f0 con 0x7ff12413c220 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 10 monclient: dump: 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff129f58640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff129f58640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff129f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0fc005090 con 0x7ff12413c220 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff114004910 con 0x7ff12413c220 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(169..169 src has 1..169) ==== 5330+0+0 (unknown 1338376868 0 0) 0x7ff11404b730 con 0x7ff12413c220 2024-09-15T20:35:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 10 client.84179.objecter ms_dispatch 0x7ff124000ba0 osd_map(169..169 src has 1..169) 2024-09-15T20:35:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 3 client.84179.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T20:35:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 3 client.84179.objecter handle_osd_map decoding full epoch 169 2024-09-15T20:35:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 20 client.84179.objecter dump_active .. 0 homeless 2024-09-15T20:35:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff1237fe640 10 monclient: get_auth_request con 0x7ff10803e060 auth_method 0 2024-09-15T20:35:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.993+0000 7ff11bfff640 10 client.84179.objecter ms_handle_connect 0x7ff10803e060 2024-09-15T20:35:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.997+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7ff114016aa0 con 0x7ff12413c220 2024-09-15T20:35:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.997+0000 7ff11bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:34.997+0000 7ff11bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:35.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:35:35.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:35.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:35.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff0fc005380 con 0x7ff12413c220 2024-09-15T20:35:35.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff11bfff640 1 -- 172.21.15.78:0/2403229891 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3474722606) 0x7ff1140163a0 con 0x7ff12413c220 2024-09-15T20:35:35.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff11bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:35.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff11bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:35.235 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:35:35.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 10 librados: watch_flush enter 2024-09-15T20:35:35.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 10 librados: watch_flush exit 2024-09-15T20:35:35.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 20 client.84179.objecter shutdown clearing up homeless session... 2024-09-15T20:35:35.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 10 client.84179.objecter successfully canceled tick 2024-09-15T20:35:35.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 >> v1:172.21.15.146:6812/435598642 conn(0x7ff10803e060 legacy=0x7ff108040520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:35.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 10 monclient: shutdown 2024-09-15T20:35:35.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:35.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.229+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 >> v1:172.21.15.78:6790/0 conn(0x7ff12413c220 legacy=0x7ff1241d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:35.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.233+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 shutdown_connections 2024-09-15T20:35:35.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.233+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 >> 172.21.15.78:0/2403229891 conn(0x7ff1240a63c0 msgr2=0x7ff12412eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:35.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.233+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 shutdown_connections 2024-09-15T20:35:35.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.233+0000 7ff129f58640 1 -- 172.21.15.78:0/2403229891 wait complete. 2024-09-15T20:35:35.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:35.233+0000 7ff129f58640 1 librados: shutdown 2024-09-15T20:35:38.254 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:35:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.376+0000 7f04638d2640 10 monclient: get_monmap_and_config 2024-09-15T20:35:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.376+0000 7f04638d2640 10 monclient: build_initial_monmap 2024-09-15T20:35:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.376+0000 7f04638d2640 10 monclient: monmap: 2024-09-15T20:35:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:38.384 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:38.384176+0000 2024-09-15T20:35:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:38.384176+0000 2024-09-15T20:35:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:38.385 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:38.385 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 1 Processor -- start 2024-09-15T20:35:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 1 -- start start 2024-09-15T20:35:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 10 monclient: init 2024-09-15T20:35:38.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 10 monclient(hunting): picked mon.noname-c con 0x7f045c134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 10 monclient(hunting): picked mon.noname-b con 0x7f045c09f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 10 monclient(hunting): picked mon.noname-a con 0x7f045c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c0a0470 con 0x7f045c06e960 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c09feb0 con 0x7f045c134bb0 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c09e4a0 con 0x7f045c09f5a0 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f04638d2640 10 monclient(hunting): authenticate will time out at 1399.269287s 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f0461e6f640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f045c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f0460e6d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f045c09f5a0 0x7f045c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38506/0 (socket says 172.21.15.78:38506) 2024-09-15T20:35:38.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.380+0000 7f0460e6d640 1 -- 172.21.15.78:0/4210876724 learned_addr learned my addr 172.21.15.78:0/4210876724 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.584+0000 7f0461e6f640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f045c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:38.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:38.988+0000 7f0461e6f640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f045c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:39.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:39.792+0000 7f0461e6f640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f045c13d0f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 10 monclient(hunting): tick 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 1 monclient(hunting): continuing hunt 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6790/0 conn(0x7f045c09f5a0 legacy=0x7f045c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.146:6789/0 conn(0x7f045c134bb0 legacy=0x7f045c134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f045c13d0f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:35:41.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f045c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:41.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0440006e60 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:41.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f045c09f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:41.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c0a0ff0 con 0x7f045c06e960 2024-09-15T20:35:41.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f044000f320 con 0x7f0440006e60 2024-09-15T20:35:41.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f046166e640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:41.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04400105c0 con 0x7f045c09f5a0 2024-09-15T20:35:41.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.384+0000 7f0452ffd640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:41.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.588+0000 7f046166e640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:41.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:41.992+0000 7f046166e640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:42.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:42.800+0000 7f046166e640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:44.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 10 monclient(hunting): tick 2024-09-15T20:35:44.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:35:44.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 1 monclient(hunting): continuing hunt 2024-09-15T20:35:44.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:35:44.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6790/0 conn(0x7f045c09f5a0 legacy=0x7f044000cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:44.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.146:6789/0 conn(0x7f0440006e60 legacy=0x7f04400092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:44.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 10 monclient(hunting): picked mon.noname-a con 0x7f045c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 10 monclient(hunting): picked mon.noname-b con 0x7f044000f620 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 10 monclient(hunting): picked mon.noname-c con 0x7f0440010790 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0440013a70 con 0x7f045c06e960 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c0a0ff0 con 0x7f0440010790 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 1 -- 172.21.15.78:0/4210876724 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0440011280 con 0x7f044000f620 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f0452ffd640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:44.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.384+0000 7f046166e640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:44.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.588+0000 7f046166e640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:44.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:44.992+0000 7f046166e640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1099566118 0 0) 0x7f0440011280 con 0x7f044000f620 2024-09-15T20:35:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient(hunting): _negotiate 2024-09-15T20:35:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:45.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient(hunting): my global_id is 84182 2024-09-15T20:35:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0448003530 con 0x7f044000f620 2024-09-15T20:35:45.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 927272234 0 0) 0x7f0448003530 con 0x7f044000f620 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.146:6789/0 conn(0x7f0440010790 legacy=0x7f0440010b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6789/0 conn(0x7f045c06e960 legacy=0x7f0440005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: _finish_hunting 0 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 monclient: found mon.noname-b 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04400114a0 con 0x7f044000f620 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: _finish_auth 0 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: _check_auth_tickets 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f044c001560 con 0x7f044000f620 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:45.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: dump: 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 5 monclient: authenticate success, global_id 84182 2024-09-15T20:35:45.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f044c001700 con 0x7f044000f620 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 1 -- 172.21.15.78:0/4210876724 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f044c001b90 con 0x7f044000f620 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 4 set_mon_vals no callback set 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0453fff640 10 monclient: dump: 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:45.666 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:45.667 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:45.667 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:45.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: shutdown 2024-09-15T20:35:45.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:45.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- 172.21.15.78:0/4210876724 >> v1:172.21.15.78:6790/0 conn(0x7f044000f620 legacy=0x7f044000faa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:45.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- 172.21.15.78:0/4210876724 shutdown_connections 2024-09-15T20:35:45.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- 172.21.15.78:0/4210876724 >> 172.21.15.78:0/4210876724 conn(0x7f045c0a63c0 msgr2=0x7f045c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- 172.21.15.78:0/4210876724 shutdown_connections 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- 172.21.15.78:0/4210876724 wait complete. 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: build_initial_monmap 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: monmap: 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:45.672 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 librados: starting msgr at 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 librados: starting objecter 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 Processor -- start 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- start start 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 librados: setting wanted keys 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 librados: calling monclient init 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: init 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient(hunting): picked mon.noname-a con 0x7f0440010790 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient(hunting): picked mon.noname-c con 0x7f045c06e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient(hunting): picked mon.noname-b con 0x7f045c134330 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:45.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c073c10 con 0x7f0440010790 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c09d180 con 0x7f045c134330 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f045c1262f0 con 0x7f045c06e960 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient(hunting): authenticate will time out at 1406.550183s 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f046166e640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f0440010790 legacy=0x7f045c1cca00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0460e6d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f045c06e960 0x7f045c133c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41716/0 (socket says 172.21.15.78:41716) 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0460e6d640 1 -- 172.21.15.78:0/3590379742 learned_addr learned my addr 172.21.15.78:0/3590379742 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 client.?.objecter ms_handle_connect 0x7f045c06e960 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 client.?.objecter ms_handle_connect 0x7f045c134330 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1134106868 0 0) 0x7f045c1262f0 con 0x7f045c06e960 2024-09-15T20:35:45.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient(hunting): my global_id is 84185 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f044000faa0 con 0x7f045c06e960 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1883710615 0 0) 0x7f044000faa0 con 0x7f045c06e960 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f045c1262f0 con 0x7f045c06e960 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f044c0013c0 con 0x7f045c06e960 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient(hunting): dump: 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:45.675 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2062916776 0 0) 0x7f045c1262f0 con 0x7f045c06e960 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 >> v1:172.21.15.146:6789/0 conn(0x7f045c134330 legacy=0x7f045c1d4d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 >> v1:172.21.15.78:6789/0 conn(0x7f0440010790 legacy=0x7f045c1cca00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 monclient: found mon.c 2024-09-15T20:35:45.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f045c128650 con 0x7f045c06e960 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient: _finish_auth 0 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 5 monclient: authenticate success, global_id 84185 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: _renew_subs 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f045c127db0 con 0x7f045c06e960 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 client.84185.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: _renew_subs 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f045c1d64a0 con 0x7f045c06e960 2024-09-15T20:35:45.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04638d2640 1 librados: init done 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f044c004aa0 con 0x7f045c06e960 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f044c004df0 con 0x7f045c06e960 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f0451ffb640 10 monclient: dump: 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:45.678 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.660+0000 7f04537fe640 4 set_mon_vals no callback set 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f04638d2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f04638d2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f04638d2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f043c005090 con 0x7f045c06e960 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f044c010e60 con 0x7f045c06e960 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(169..169 src has 1..169) ==== 5330+0+0 (unknown 1338376868 0 0) 0x7f044c04b530 con 0x7f045c06e960 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f0451ffb640 10 client.84185.objecter ms_dispatch 0x7f045c000ba0 osd_map(169..169 src has 1..169) 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f0451ffb640 3 client.84185.objecter handle_osd_map got epochs [169,169] > 0 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f0451ffb640 3 client.84185.objecter handle_osd_map decoding full epoch 169 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f0451ffb640 20 client.84185.objecter dump_active .. 0 homeless 2024-09-15T20:35:45.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f046166e640 10 monclient: get_auth_request con 0x7f044003f6a0 auth_method 0 2024-09-15T20:35:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.664+0000 7f0451ffb640 10 client.84185.objecter ms_handle_connect 0x7f044003f6a0 2024-09-15T20:35:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.668+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f044c0168a0 con 0x7f045c06e960 2024-09-15T20:35:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.668+0000 7f0451ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:45.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.668+0000 7f0451ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:35:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f043c005380 con 0x7f045c06e960 2024-09-15T20:35:45.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f0451ffb640 1 -- 172.21.15.78:0/3590379742 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 1781078884) 0x7f044c01faa0 con 0x7f045c06e960 2024-09-15T20:35:45.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f0451ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:35:45.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f0451ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:45.911 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":20,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:35:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 10 librados: watch_flush enter 2024-09-15T20:35:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 10 librados: watch_flush exit 2024-09-15T20:35:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 20 client.84185.objecter shutdown clearing up homeless session... 2024-09-15T20:35:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 10 client.84185.objecter successfully canceled tick 2024-09-15T20:35:45.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 >> v1:172.21.15.146:6812/435598642 conn(0x7f044003f6a0 legacy=0x7f0440041b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 10 monclient: shutdown 2024-09-15T20:35:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 >> v1:172.21.15.78:6790/0 conn(0x7f045c06e960 legacy=0x7f045c133c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 shutdown_connections 2024-09-15T20:35:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.904+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 >> 172.21.15.78:0/3590379742 conn(0x7f045c0a63c0 msgr2=0x7f045c06ed40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.908+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 shutdown_connections 2024-09-15T20:35:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.908+0000 7f04638d2640 1 -- 172.21.15.78:0/3590379742 wait complete. 2024-09-15T20:35:45.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:45.908+0000 7f04638d2640 1 librados: shutdown 2024-09-15T20:35:45.929 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:35:45.929 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T20:35:45.929 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: get_monmap_and_config 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: build_initial_monmap 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: monmap: 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:46.049044+0000 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:46.049044+0000 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:46.049 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:46.050 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:46.050 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 Processor -- start 2024-09-15T20:35:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- start start 2024-09-15T20:35:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: init 2024-09-15T20:35:46.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient(hunting): picked mon.noname-b con 0x7f1804137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient(hunting): picked mon.noname-a con 0x7f18040a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient(hunting): picked mon.noname-c con 0x7f180413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18040a6420 con 0x7f18040a9610 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18040a6700 con 0x7f180413e5f0 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18040a2110 con 0x7f1804137ce0 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient(hunting): authenticate will time out at 1406.933567s 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1802ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1804137ce0 0x7f18041380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41730/0 (socket says 172.21.15.78:41730) 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1802ffd640 1 -- 172.21.15.78:0/1006594715 learned_addr learned my addr 172.21.15.78:0/1006594715 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18027fc640 1 -- 172.21.15.78:0/1006594715 >> v1:172.21.15.78:6789/0 conn(0x7f18040a9610 legacy=0x7f180413ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:46.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 384187076 0 0) 0x7f18040a2110 con 0x7f1804137ce0 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient(hunting): my global_id is 84188 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17ec003530 con 0x7f1804137ce0 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 240508714 0 0) 0x7f17ec003530 con 0x7f1804137ce0 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 >> v1:172.21.15.146:6789/0 conn(0x7f180413e5f0 legacy=0x7f18041409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 >> v1:172.21.15.78:6789/0 conn(0x7f18040a9610 legacy=0x7f180413ced0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 monclient: found mon.noname-b 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18040a4bc0 con 0x7f1804137ce0 2024-09-15T20:35:46.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: _finish_auth 0 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 5 monclient: authenticate success, global_id 84188 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17f40016a0 con 0x7f1804137ce0 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: dump: 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.060 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17f4001900 con 0x7f1804137ce0 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 1 -- 172.21.15.78:0/1006594715 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17f4001d10 con 0x7f1804137ce0 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 4 set_mon_vals no callback set 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:46.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f1801ffb640 10 monclient: dump: 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: shutdown 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- 172.21.15.78:0/1006594715 >> v1:172.21.15.78:6790/0 conn(0x7f1804137ce0 legacy=0x7f18041380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- 172.21.15.78:0/1006594715 shutdown_connections 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- 172.21.15.78:0/1006594715 >> 172.21.15.78:0/1006594715 conn(0x7f18040aa6c0 msgr2=0x7f18040aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- 172.21.15.78:0/1006594715 shutdown_connections 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- 172.21.15.78:0/1006594715 wait complete. 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: build_initial_monmap 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: monmap: 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:46.063 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 librados: starting msgr at 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 librados: starting objecter 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 Processor -- start 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 -- start start 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 librados: setting wanted keys 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 1 librados: calling monclient init 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.044+0000 7f18092e7640 10 monclient: init 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:35:46.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient(hunting): picked mon.noname-a con 0x7f18040a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient(hunting): picked mon.noname-b con 0x7f1804137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient(hunting): picked mon.noname-c con 0x7f180413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1804141b20 con 0x7f18040a9610 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f180413bea0 con 0x7f1804137ce0 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1804139cd0 con 0x7f180413e5f0 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f1802ffd640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f18040a9610 legacy=0x7f1804138eb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient(hunting): authenticate will time out at 1406.936452s 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18037fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f180413e5f0 0x7f18041d4f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41738/0 (socket says 172.21.15.78:41738) 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18037fe640 1 -- 172.21.15.78:0/2414895749 learned_addr learned my addr 172.21.15.78:0/2414895749 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 client.?.objecter ms_handle_connect 0x7f180413e5f0 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:46.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 546950998 0 0) 0x7f1804139cd0 con 0x7f180413e5f0 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): my global_id is 84191 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17f0004ec0 con 0x7f180413e5f0 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 478057853 0 0) 0x7f17f0004ec0 con 0x7f180413e5f0 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1804139cd0 con 0x7f180413e5f0 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 client.?.objecter ms_handle_connect 0x7f1804137ce0 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17fc004390 con 0x7f180413e5f0 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): dump: 2024-09-15T20:35:46.066 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1762488071 0 0) 0x7f1804139cd0 con 0x7f180413e5f0 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 >> v1:172.21.15.146:6789/0 conn(0x7f1804137ce0 legacy=0x7f18041395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 >> v1:172.21.15.78:6789/0 conn(0x7f18040a9610 legacy=0x7f1804138eb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 monclient: found mon.c 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f180413db80 con 0x7f180413e5f0 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: _finish_auth 0 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 5 monclient: authenticate success, global_id 84191 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient: _renew_subs 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f18040a14b0 con 0x7f180413e5f0 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 client.84191.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:46.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient: _renew_subs 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f180413bbc0 con 0x7f180413e5f0 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 1 librados: init done 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f17fc004730 con 0x7f180413e5f0 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17fc0016b0 con 0x7f180413e5f0 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18017fa640 4 set_mon_vals no callback set 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: dump: 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.069 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f17e40065d0 con 0x7f17e4002870 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 client.84191.objecter ms_handle_connect 0x7f17e4002870 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 client.84191.objecter resend_mon_ops 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f17fc001850 con 0x7f180413e5f0 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(170..170 src has 1..170) ==== 5606+0+0 (unknown 555610031 0 0) 0x7f17fc04bbf0 con 0x7f180413e5f0 2024-09-15T20:35:46.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 client.84191.objecter ms_dispatch 0x7f1804000ba0 osd_map(170..170 src has 1..170) 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 3 client.84191.objecter handle_osd_map got epochs [170,170] > 0 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 3 client.84191.objecter handle_osd_map decoding full epoch 170 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 20 client.84191.objecter dump_active .. 0 homeless 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1013736882 0 0) 0x7f17e40065d0 con 0x7f17e4002870 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient(hunting): my global_id is 84194 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1804139cd0 con 0x7f17e4002870 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f18027fc640 10 monclient: get_auth_request con 0x7f17f003ded0 auth_method 0 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2283607121 0 0) 0x7f1804139cd0 con 0x7f17e4002870 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17e40065d0 con 0x7f17e4002870 2024-09-15T20:35:46.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17f4001890 con 0x7f17e4002870 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: dump: 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3739712748 0 0) 0x7f17e40065d0 con 0x7f17e4002870 2024-09-15T20:35:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f17e400a170 con 0x7f17e4002870 2024-09-15T20:35:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 client.84191.objecter ms_handle_connect 0x7f17f003ded0 2024-09-15T20:35:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f17f40016f0 con 0x7f17e4002870 2024-09-15T20:35:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 >> v1:172.21.15.78:6790/0 conn(0x7f17e4002870 legacy=0x7f17e4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.048+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 >> v1:172.21.15.78:6790/0 conn(0x7f17e4002870 legacy=0x7f17e4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:46.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:35:46.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f17e4009e60 con 0x7f17e4009120 2024-09-15T20:35:46.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 client.84191.objecter ms_handle_connect 0x7f17e4009120 2024-09-15T20:35:46.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 client.84191.objecter resend_mon_ops 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2309508241 0 0) 0x7f17e4009e60 con 0x7f17e4009120 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient(hunting): my global_id is 84197 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f17e40065d0 con 0x7f17e4009120 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2724599866 0 0) 0x7f17e40065d0 con 0x7f17e4009120 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f17e4009e60 con 0x7f17e4009120 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f17f4004b70 con 0x7f17e4009120 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient: dump: 2024-09-15T20:35:46.075 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 909978307 0 0) 0x7f17e4009e60 con 0x7f17e4009120 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f17e400c400 con 0x7f17e4009120 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4126 (unknown 0 0 1303386052) 0x7f17f4004b60 con 0x7f17e4009120 2024-09-15T20:35:46.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 >> v1:172.21.15.78:6790/0 conn(0x7f17e4009120 legacy=0x7f17e40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f17eb7fe640 1 -- 172.21.15.78:0/2414895749 >> v1:172.21.15.78:6790/0 conn(0x7f17e4009120 legacy=0x7f17e40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 20, 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17743, 2024-09-15T20:35:46.077 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:46.078 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:46.079 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:35:46.080 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:46.081 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:35:46.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:35:46.083 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:46.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:46.085 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.086 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:46.087 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:35:46.088 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:35:46.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 10 librados: watch_flush enter 2024-09-15T20:35:46.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 10 librados: watch_flush exit 2024-09-15T20:35:46.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 20 client.84191.objecter shutdown clearing up homeless session... 2024-09-15T20:35:46.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 10 client.84191.objecter successfully canceled tick 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 >> v1:172.21.15.146:6812/435598642 conn(0x7f17f003ded0 legacy=0x7f17f0040390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 10 monclient: shutdown 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 >> v1:172.21.15.78:6790/0 conn(0x7f180413e5f0 legacy=0x7f18041d4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18037fe640 1 -- 172.21.15.78:0/2414895749 reap_dead start 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18037fe640 1 -- 172.21.15.78:0/2414895749 reap_dead start 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 shutdown_connections 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 >> 172.21.15.78:0/2414895749 conn(0x7f18040aa6c0 msgr2=0x7f180413e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 shutdown_connections 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 1 -- 172.21.15.78:0/2414895749 wait complete. 2024-09-15T20:35:46.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.056+0000 7f18092e7640 1 librados: shutdown 2024-09-15T20:35:46.092 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:35:46.092 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T20:35:46.092 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: get_monmap_and_config 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: build_initial_monmap 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: monmap: 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:35:46.207944+0000 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:35:46.207944+0000 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:46.208 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.209 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:46.209 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:35:46.209 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:35:46.209 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 Processor -- start 2024-09-15T20:35:46.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- start start 2024-09-15T20:35:46.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: init 2024-09-15T20:35:46.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:46.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:35:46.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb8135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:46.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb80a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:46.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb813c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb80745e0 con 0x7f2fb813c220 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb806ecf0 con 0x7f2fb80a24e0 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb806efd0 con 0x7f2fb8135b60 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): authenticate will time out at 1407.092469s 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbcba7640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f2fb813c220 legacy=0x7f2fb813e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb7fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f2fb8135b60 0x7f2fb8135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41754/0 (socket says 172.21.15.78:41754) 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb7fff640 1 -- 172.21.15.78:0/579446992 learned_addr learned my addr 172.21.15.78:0/579446992 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083537065 0 0) 0x7f2fb806efd0 con 0x7f2fb8135b60 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:46.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient(hunting): my global_id is 84200 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fa0003530 con 0x7f2fb8135b60 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 157730105 0 0) 0x7f2fa0003530 con 0x7f2fb8135b60 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 >> v1:172.21.15.146:6789/0 conn(0x7f2fb80a24e0 legacy=0x7f2fb813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 >> v1:172.21.15.78:6789/0 conn(0x7f2fb813c220 legacy=0x7f2fb813e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 monclient: found mon.noname-b 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb8136650 con 0x7f2fb8135b60 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: _finish_auth 0 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:46.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 5 monclient: authenticate success, global_id 84200 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2fa80016a0 con 0x7f2fb8135b60 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: dump: 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.220 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2fa80018c0 con 0x7f2fb8135b60 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 1 -- 172.21.15.78:0/579446992 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2fa8001cd0 con 0x7f2fb8135b60 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: get_monmap_and_config success 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 4 set_mon_vals no callback set 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb6ffd640 10 monclient: dump: 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.221 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: shutdown 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- 172.21.15.78:0/579446992 >> v1:172.21.15.78:6790/0 conn(0x7f2fb8135b60 legacy=0x7f2fb8135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- 172.21.15.78:0/579446992 shutdown_connections 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- 172.21.15.78:0/579446992 >> 172.21.15.78:0/579446992 conn(0x7f2fb80a63c0 msgr2=0x7f2fb80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- 172.21.15.78:0/579446992 shutdown_connections 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- 172.21.15.78:0/579446992 wait complete. 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: build_initial_monmap 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: monmap: 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:35:46.223 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 librados: starting msgr at 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 librados: starting objecter 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 Processor -- start 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- start start 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 librados: setting wanted keys 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 librados: calling monclient init 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: init 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: _reopen_session rank -1 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): picked mon.noname-c con 0x7f2fb80a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:35:46.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): picked mon.noname-a con 0x7f2fb8135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): picked mon.noname-b con 0x7f2fb813c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb813bcc0 con 0x7f2fb8135b60 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb8138010 con 0x7f2fb813c220 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fb809d180 con 0x7f2fb80a24e0 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): _renew_subs 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb77fe640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f2fb8135b60 legacy=0x7f2fb8072730 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fbe60a640 10 monclient(hunting): authenticate will time out at 1407.095326s 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb7fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f2fb80a24e0 0x7f2fb8072020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41758/0 (socket says 172.21.15.78:41758) 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb7fff640 1 -- 172.21.15.78:0/191334018 learned_addr learned my addr 172.21.15.78:0/191334018 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 client.?.objecter ms_handle_connect 0x7f2fb80a24e0 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2729057039 0 0) 0x7f2fb809d180 con 0x7f2fb80a24e0 2024-09-15T20:35:46.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 monclient(hunting): my global_id is 84203 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fa4004ec0 con 0x7f2fb80a24e0 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 client.?.objecter ms_handle_connect 0x7f2fb813c220 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2536671364 0 0) 0x7f2fa4004ec0 con 0x7f2fb80a24e0 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2fb809d180 con 0x7f2fb80a24e0 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2fa8001c10 con 0x7f2fb80a24e0 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 monclient(hunting): dump: 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.226 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3200072863 0 0) 0x7f2fb809d180 con 0x7f2fb80a24e0 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.204+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 >> v1:172.21.15.146:6789/0 conn(0x7f2fb813c220 legacy=0x7f2fb81d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 >> v1:172.21.15.78:6789/0 conn(0x7f2fb8135b60 legacy=0x7f2fb8072730 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:35:46.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 monclient: found mon.c 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fb8073740 con 0x7f2fb80a24e0 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: _finish_auth 0 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 5 monclient: authenticate success, global_id 84203 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 10 monclient: _renew_subs 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2fb8138910 con 0x7f2fb80a24e0 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 10 client.84203.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 10 monclient: _renew_subs 2024-09-15T20:35:46.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2fa8004c40 con 0x7f2fb80a24e0 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2fb8072e40 con 0x7f2fb80a24e0 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 1 librados: init done 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2fa8004fd0 con 0x7f2fb80a24e0 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: dump: 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.229 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb67fc640 4 set_mon_vals no callback set 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2fb8073180 con 0x7f2fb813cd60 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f2fa80110c0 con 0x7f2fb80a24e0 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(170..170 src has 1..170) ==== 5606+0+0 (unknown 555610031 0 0) 0x7f2fa804b9a0 con 0x7f2fb80a24e0 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 client.84203.objecter ms_dispatch 0x7f2fb8000ba0 osd_map(170..170 src has 1..170) 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 3 client.84203.objecter handle_osd_map got epochs [170,170] > 0 2024-09-15T20:35:46.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 3 client.84203.objecter handle_osd_map decoding full epoch 170 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 20 client.84203.objecter dump_active .. 0 homeless 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 client.84203.objecter ms_handle_connect 0x7f2fb813cd60 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 client.84203.objecter resend_mon_ops 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fbcba7640 10 monclient: get_auth_request con 0x7f2fa403def0 auth_method 0 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3350835094 0 0) 0x7f2fb8073180 con 0x7f2fb813cd60 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient(hunting): my global_id is 23254 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fb809d180 con 0x7f2fb813cd60 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 client.84203.objecter ms_handle_connect 0x7f2fa403def0 2024-09-15T20:35:46.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1978924393 0 0) 0x7f2fb809d180 con 0x7f2fb813cd60 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2fb8073180 con 0x7f2fb813cd60 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2fac001780 con 0x7f2fb813cd60 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: dump: 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.232 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2660318688 0 0) 0x7f2fb8073180 con 0x7f2fb813cd60 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2fb80008e0 con 0x7f2fb813cd60 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2fac001a80 con 0x7f2fb813cd60 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 >> v1:172.21.15.146:6789/0 conn(0x7f2fb813cd60 legacy=0x7f2fb813d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.208+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 >> v1:172.21.15.146:6789/0 conn(0x7f2fb813cd60 legacy=0x7f2fb813d140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fbe60a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2fb813dc80 con 0x7f2fb813d850 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 client.84203.objecter ms_handle_connect 0x7f2fb813d850 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 client.84203.objecter resend_mon_ops 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33440405 0 0) 0x7f2fb813dc80 con 0x7f2fb813d850 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient(hunting): my global_id is 23257 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fb8073180 con 0x7f2fb813d850 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1111951099 0 0) 0x7f2fb8073180 con 0x7f2fb813d850 2024-09-15T20:35:46.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2fb813dc80 con 0x7f2fb813d850 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2fac003ad0 con 0x7f2fb813d850 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient: dump: 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:35:46.236 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:35:46.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3914319133 0 0) 0x7f2fb813dc80 con 0x7f2fb813d850 2024-09-15T20:35:46.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2fb81357e0 con 0x7f2fb813d850 2024-09-15T20:35:46.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 270910823) 0x7f2fac0057d0 con 0x7f2fb813d850 2024-09-15T20:35:46.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:35:46.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:35:46.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 >> v1:172.21.15.146:6789/0 conn(0x7f2fb813d850 legacy=0x7f2fb81d8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.216+0000 7f2fb4ff9640 1 -- 172.21.15.78:0/191334018 >> v1:172.21.15.146:6789/0 conn(0x7f2fb813d850 legacy=0x7f2fb81d8350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:35:46.237 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 20, 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 258733, 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:35:46.238 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:46.239 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:46.240 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:35:46.241 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:35:46.242 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:46.243 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.244 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.245 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.246 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.247 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.248 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 8 2024-09-15T20:35:46.249 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:35:46.250 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 10 librados: watch_flush enter 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 10 librados: watch_flush exit 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 20 client.84203.objecter shutdown clearing up homeless session... 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 10 client.84203.objecter successfully canceled tick 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 >> v1:172.21.15.146:6812/435598642 conn(0x7f2fa403def0 legacy=0x7f2fa40403b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 10 monclient: shutdown 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 >> v1:172.21.15.78:6790/0 conn(0x7f2fb80a24e0 legacy=0x7f2fb8072020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:35:46.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbcba7640 1 -- 172.21.15.78:0/191334018 reap_dead start 2024-09-15T20:35:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbcba7640 1 -- 172.21.15.78:0/191334018 reap_dead start 2024-09-15T20:35:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 shutdown_connections 2024-09-15T20:35:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 >> 172.21.15.78:0/191334018 conn(0x7f2fb80a63c0 msgr2=0x7f2fb80a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:35:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 shutdown_connections 2024-09-15T20:35:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 1 -- 172.21.15.78:0/191334018 wait complete. 2024-09-15T20:35:46.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:35:46.220+0000 7f2fbe60a640 1 librados: shutdown 2024-09-15T20:35:46.253 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:35:46.253 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:36:26.633 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleWrite (7026 ms) 2024-09-15T20:36:26.633 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.WaitForComplete 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.WaitForComplete (6968 ms) 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip (7061 ms) 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTrip2 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTrip2 (6946 ms) 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripAppend 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripAppend (6994 ms) 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.IsComplete 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.IsComplete (6999 ms) 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.IsSafe 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.IsSafe (7074 ms) 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.ReturnValue 2024-09-15T20:36:26.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.ReturnValue (6962 ms) 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.Flush 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.Flush (6967 ms) 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.FlushAsync 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.FlushAsync (7048 ms) 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.RoundTripWriteFull 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.RoundTripWriteFull (20987 ms) 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStat 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStat (7006 ms) 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.SimpleStatNS 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.SimpleStatNS (6992 ms) 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.StatRemove 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.StatRemove (6988 ms) 2024-09-15T20:36:26.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.ExecuteClass 2024-09-15T20:36:26.636 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.ExecuteClass (7017 ms) 2024-09-15T20:36:26.636 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ RUN ] LibRadosAioEC.MultiWrite 2024-09-15T20:36:26.636 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ OK ] LibRadosAioEC.MultiWrite (7015 ms) 2024-09-15T20:36:26.636 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [----------] 16 tests from LibRadosAioEC (126051 ms total) 2024-09-15T20:36:26.636 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: 2024-09-15T20:36:26.636 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [----------] Global test environment tear-down 2024-09-15T20:36:26.636 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [==========] 42 tests from 2 test suites ran. (218035 ms total) 2024-09-15T20:36:26.636 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio: [ PASSED ] 42 tests. 2024-09-15T20:36:30.650 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [==========] Running 77 tests from 4 test suites. 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] Global test environment set-up. 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: seed 28320 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.Dirty 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTierPP.Dirty (214 ms) 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.FlushWriteRaces 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTierPP.FlushWriteRaces (10840 ms) 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTierPP.HitSetNone 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTierPP.HitSetNone (110 ms) 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] 3 tests from LibRadosTierPP (11164 ms total) 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: 2024-09-15T20:36:30.651 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Overlay 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Overlay (7044 ms) 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Promote 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Promote (7938 ms) 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnap 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnap (10015 ms) 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapScrub 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: my_snaps [3] 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: my_snaps [4,3] 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: my_snaps [5,4,3] 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: my_snaps [6,5,4,3] 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: promoting some heads 2024-09-15T20:36:30.652 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: promoting from clones for snap 6 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: promoting from clones for snap 5 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: promoting from clones for snap 4 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: promoting from clones for snap 3 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: waiting for scrubs... 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: done waiting 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapScrub (47034 ms) 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteSnapTrimRace 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteSnapTrimRace (10006 ms) 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Whiteout 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Whiteout (7990 ms) 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.WhiteoutDeleteCreate (8070 ms) 2024-09-15T20:36:30.653 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Evict 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Evict (7951 ms) 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap (10012 ms) 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnap2 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnap2 (9025 ms) 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ListSnap 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ListSnap (10019 ms) 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.EvictSnapRollbackReadRace (27123 ms) 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlush 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlush (7853 ms) 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.Flush 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.Flush (8021 ms) 2024-09-15T20:36:30.654 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushSnap 2024-09-15T20:36:30.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushSnap (12984 ms) 2024-09-15T20:36:30.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.FlushTryFlushRaces 2024-09-15T20:36:30.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.FlushTryFlushRaces (7982 ms) 2024-09-15T20:36:30.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TryFlushReadRace 2024-09-15T20:36:30.655 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TryFlushReadRace (8067 ms) 2024-09-15T20:36:46.324 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetRead 2024-09-15T20:36:46.324 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: ok, hit_set contains 267:602f83fe:::foo:head 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetRead (8969 ms) 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetWrite 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg_num = 32 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 0 ls 1726432606,0 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 1 ls 1726432606,0 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 2 ls 1726432606,0 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 3 ls 1726432606,0 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 4 ls 1726432606,0 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 5 ls 1726432606,0 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 6 ls 1726432606,0 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 7 ls 1726432606,0 2024-09-15T20:36:46.325 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 8 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 9 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 10 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 11 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 12 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 13 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 14 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 15 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 16 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 17 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 18 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 19 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 20 ls 1726432606,0 2024-09-15T20:36:46.326 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 21 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 22 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 23 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 24 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 25 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 26 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 27 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 28 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 29 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 30 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg 31 ls 1726432606,0 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: pg_num = 32 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:6cac518f:::0:head 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:02547ec2:::1:head 2024-09-15T20:36:46.327 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f905c69b:::2:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:cfc208b3:::3:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d83876eb:::4:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b29083e3:::5:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c4fdafeb:::6:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:5c6b0b28:::7:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:bd63b0f1:::8:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:e960b815:::9:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:52ea6a34:::10:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:89d3ae78:::11:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:de5d7c5f:::12:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:566253c9:::13:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:62a1935d:::14:head 2024-09-15T20:36:46.328 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:863748b0:::15:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:3958e169:::16:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:4d4dabf9:::17:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:8391935d:::18:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:28883081:::19:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:69259c59:::20:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:4bdb80b7:::21:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a11c5d71:::22:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:271af37b:::23:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:95b121be:::24:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:58d1031b:::25:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:0a050783:::26:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c709704c:::27:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:cbe56eaf:::28:head 2024-09-15T20:36:46.329 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:86b4b162:::29:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:70d89383:::30:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:dd450c7c:::31:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:6d5729b1:::32:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c388f3fb:::33:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:56cfea31:::34:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:9dbc1bf7:::35:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:40b74ccd:::36:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:4d5aaf42:::37:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:920f362c:::38:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:6cc53222:::39:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:9cad833f:::40:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:1ea84d41:::41:head 2024-09-15T20:36:46.330 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c4480ef6:::42:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a694361e:::43:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d1bd33e9:::44:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:ddc2cd5d:::45:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:2b782207:::46:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:7b187fca:::47:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:90ecdf6f:::48:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a5ed95fe:::49:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:ea0eaa55:::50:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f33ef17b:::51:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a0d1b2f6:::52:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:60c5229e:::53:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:edcbc575:::54:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:102cf253:::55:head 2024-09-15T20:36:46.331 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:efb7fb0b:::56:head 2024-09-15T20:36:46.332 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:50d0a326:::57:head 2024-09-15T20:36:46.332 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d4dc5daf:::58:head 2024-09-15T20:36:46.332 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:3a130462:::59:head 2024-09-15T20:36:46.332 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:ec87ed71:::60:head 2024-09-15T20:36:46.332 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d5bc9454:::61:head 2024-09-15T20:36:46.332 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:3ddfe313:::62:head 2024-09-15T20:36:46.332 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:7c2816b9:::63:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:47e00e4d:::64:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c6410c18:::65:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b48ed237:::66:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:cd63ad31:::67:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b179e92b:::68:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:0d9f741a:::69:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:6d3352ae:::70:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c6d5c19e:::71:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:bc4729c3:::72:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:77e930b9:::73:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:0abeecfd:::74:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b7c37e15:::75:head 2024-09-15T20:36:46.333 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b6378398:::76:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:02bd68de:::77:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:cc795d2d:::78:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:630d4fea:::79:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:e0d29ef5:::80:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:fd6f13d2:::81:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:606461d5:::82:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:eadbdc43:::83:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:8761d0bb:::84:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:9ef0186f:::85:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:e0d41294:::86:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:961de695:::87:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:1423148f:::88:head 2024-09-15T20:36:46.334 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:633a8fa2:::89:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a8653809:::90:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:3dac8b33:::91:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:35aad435:::92:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f6dcc343:::93:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:dbbdad87:::94:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:1cb48ce0:::95:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:03cd461c:::96:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:17a4ea99:::97:head 2024-09-15T20:36:46.335 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:9993c9a7:::98:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:6394211c:::99:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:94c7ae57:::100:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:6fdee5bb:::101:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:9a477fd1:::102:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:eb850916:::103:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:affc56b9:::104:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b42dc814:::105:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f319f8f0:::106:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:9a40b9de:::107:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:8b524f28:::108:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:e3de589f:::109:head 2024-09-15T20:36:46.336 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:90f90a5b:::110:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a7b4f1d7:::111:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:af51766e:::112:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b6f90bd1:::113:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:e0261208:::114:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c9569ef7:::115:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:61bebe50:::116:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:fe93412b:::117:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d3d38bee:::118:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:3100ba0c:::119:head 2024-09-15T20:36:46.337 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d0560ada:::120:head 2024-09-15T20:36:46.338 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f0ea8b35:::121:head 2024-09-15T20:36:46.338 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:766f231a:::122:head 2024-09-15T20:36:46.338 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a07a2582:::123:head 2024-09-15T20:36:46.338 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:bd7c6b3a:::124:head 2024-09-15T20:36:46.338 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:fb2ddaff:::125:head 2024-09-15T20:36:46.338 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:4408e1fe:::126:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:ee1df7a7:::127:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c3002909:::128:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:4f48ffa9:::129:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:edf38733:::130:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c08425c0:::131:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:5f902d98:::132:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:41ea2c93:::133:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:813cee13:::134:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:0131818d:::135:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:26ba5a85:::136:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:381b8a5a:::137:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:28797e47:::138:head 2024-09-15T20:36:46.339 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:bfca7f22:::139:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:36807075:::140:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:80b03975:::141:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:5c15709b:::142:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f39ea15e:::143:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:ea992956:::144:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:48887b1c:::145:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:9f24a9dd:::146:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:987f100b:::147:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d2dd3581:::148:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:7fed1808:::149:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c80b70e9:::150:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:85ed90f9:::151:head 2024-09-15T20:36:46.340 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:36428b24:::152:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d044c34a:::153:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:7c18bf58:::154:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d1c21232:::155:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a7a3c575:::156:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:87da0633:::157:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d5ac3822:::158:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:3f20522d:::159:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:6ca26563:::160:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:532ce135:::161:head 2024-09-15T20:36:46.341 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:c78863e6:::162:head 2024-09-15T20:37:16.254 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T20:37:16.255 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T20:37:16.256 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T20:37:16.260 INFO:tasks.ceph.mon.a:Started 2024-09-15T20:37:16.260 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:37:16.260 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:37:16.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient: get_monmap_and_config 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient: build_initial_monmap 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient: monmap: 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:16.381311+0000 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:16.381311+0000 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:16.381 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:16.382 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 Processor -- start 2024-09-15T20:37:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- start start 2024-09-15T20:37:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient: init 2024-09-15T20:37:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:37:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa824137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa8240a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa82413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:16.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8240a6420 con 0x7fa82413e5f0 2024-09-15T20:37:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8240a6700 con 0x7fa8240a9610 2024-09-15T20:37:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa8240a2110 con 0x7fa824137ce0 2024-09-15T20:37:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient(hunting): authenticate will time out at 1497.265914s 2024-09-15T20:37:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa823fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa824137ce0 0x7fa8241380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57194/0 (socket says 172.21.15.78:57194) 2024-09-15T20:37:16.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa823fff640 1 -- 172.21.15.78:0/3747175316 learned_addr learned my addr 172.21.15.78:0/3747175316 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3484143834 0 0) 0x7fa8240a2110 con 0x7fa824137ce0 2024-09-15T20:37:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient(hunting): my global_id is 84308 2024-09-15T20:37:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa818003530 con 0x7fa824137ce0 2024-09-15T20:37:16.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2898818950 0 0) 0x7fa818003530 con 0x7fa824137ce0 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 >> v1:172.21.15.146:6789/0 conn(0x7fa8240a9610 legacy=0x7fa82413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 >> v1:172.21.15.78:6789/0 conn(0x7fa82413e5f0 legacy=0x7fa8241409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 monclient: found mon.noname-b 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8240a4bc0 con 0x7fa824137ce0 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: _finish_auth 0 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 5 monclient: authenticate success, global_id 84308 2024-09-15T20:37:16.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa80c0016a0 con 0x7fa824137ce0 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: dump: 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.393 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa80c0018c0 con 0x7fa824137ce0 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 1 -- 172.21.15.78:0/3747175316 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa80c001c90 con 0x7fa824137ce0 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 4 set_mon_vals no callback set 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa822ffd640 10 monclient: dump: 2024-09-15T20:37:16.394 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.395 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.395 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.395 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.395 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 10 monclient: shutdown 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3747175316 >> v1:172.21.15.78:6790/0 conn(0x7fa824137ce0 legacy=0x7fa8241380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3747175316 shutdown_connections 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3747175316 >> 172.21.15.78:0/3747175316 conn(0x7fa8240aa6c0 msgr2=0x7fa8240aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3747175316 shutdown_connections 2024-09-15T20:37:16.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.380+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3747175316 wait complete. 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: build_initial_monmap 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: monmap: 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 librados: starting msgr at 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 librados: starting objecter 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 Processor -- start 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 -- start start 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 librados: setting wanted keys 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 librados: calling monclient init 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: init 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient(hunting): picked mon.noname-a con 0x7fa8240a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient(hunting): picked mon.noname-c con 0x7fa824137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient(hunting): picked mon.noname-b con 0x7fa82413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa824139090 con 0x7fa8240a9610 2024-09-15T20:37:16.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82409da90 con 0x7fa82413e5f0 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa82409ff50 con 0x7fa824137ce0 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient(hunting): authenticate will time out at 1497.269254s 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa8237fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa824137ce0 0x7fa82409d380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57196/0 (socket says 172.21.15.78:57196) 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa8237fe640 1 -- 172.21.15.78:0/3603946132 learned_addr learned my addr 172.21.15.78:0/3603946132 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 client.?.objecter ms_handle_connect 0x7fa824137ce0 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4059430973 0 0) 0x7fa82409ff50 con 0x7fa824137ce0 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): my global_id is 84311 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa804004ec0 con 0x7fa824137ce0 2024-09-15T20:37:16.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 client.?.objecter ms_handle_connect 0x7fa82413e5f0 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 client.?.objecter ms_handle_connect 0x7fa8240a9610 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1045897835 0 0) 0x7fa804004ec0 con 0x7fa824137ce0 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa82409ff50 con 0x7fa824137ce0 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa814001a40 con 0x7fa824137ce0 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): dump: 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.400 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851952607 0 0) 0x7fa82409da90 con 0x7fa82413e5f0 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient(hunting): my global_id is 23371 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa804004ec0 con 0x7fa82413e5f0 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2633632413 0 0) 0x7fa82409ff50 con 0x7fa824137ce0 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 >> v1:172.21.15.146:6789/0 conn(0x7fa82413e5f0 legacy=0x7fa8241d5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 >> v1:172.21.15.78:6789/0 conn(0x7fa8240a9610 legacy=0x7fa82409cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 monclient: found mon.c 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8240a0230 con 0x7fa824137ce0 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient: _finish_auth 0 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 5 monclient: authenticate success, global_id 84311 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: _renew_subs 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa82409fb90 con 0x7fa824137ce0 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 client.84311.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: _renew_subs 2024-09-15T20:37:16.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa82409ec10 con 0x7fa824137ce0 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 librados: init done 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa814001d00 con 0x7fa824137ce0 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa814003a70 con 0x7fa824137ce0 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 monclient: dump: 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.403 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa8227fc640 4 set_mon_vals no callback set 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fa814003c90 con 0x7fa824137ce0 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa82406a080 con 0x7fa824137ce0 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(260..260 src has 1..260) ==== 6217+0+0 (unknown 1001297475 0 0) 0x7fa81404bbe0 con 0x7fa824137ce0 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 10 client.84311.objecter ms_dispatch 0x7fa824000ba0 osd_map(260..260 src has 1..260) 2024-09-15T20:37:16.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 3 client.84311.objecter handle_osd_map got epochs [260,260] > 0 2024-09-15T20:37:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 3 client.84311.objecter handle_osd_map decoding full epoch 260 2024-09-15T20:37:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.384+0000 7fa820ff9640 20 client.84311.objecter dump_active .. 0 homeless 2024-09-15T20:37:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.388+0000 7fa823fff640 10 monclient: get_auth_request con 0x7fa80403e140 auth_method 0 2024-09-15T20:37:16.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.388+0000 7fa820ff9640 10 client.84311.objecter ms_handle_connect 0x7fa80403e140 2024-09-15T20:37:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.412+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fa814016b80 con 0x7fa824137ce0 2024-09-15T20:37:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.412+0000 7fa820ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:16.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.412+0000 7fa820ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:16.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa82a8d6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:37:16.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa82a8d6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:37:16.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa82a8d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa824076db0 con 0x7fa824137ce0 2024-09-15T20:37:16.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa820ff9640 1 -- 172.21.15.78:0/3603946132 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 4150410475) 0x7fa814016480 con 0x7fa824137ce0 2024-09-15T20:37:16.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa820ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:37:16.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa820ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:16.655 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:37:16.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa82a8d6640 10 librados: watch_flush enter 2024-09-15T20:37:16.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.652+0000 7fa82a8d6640 10 librados: watch_flush exit 2024-09-15T20:37:16.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 20 client.84311.objecter shutdown clearing up homeless session... 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 10 client.84311.objecter successfully canceled tick 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 >> v1:172.21.15.146:6812/435598642 conn(0x7fa80403e140 legacy=0x7fa804040600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 10 monclient: shutdown 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 >> v1:172.21.15.78:6790/0 conn(0x7fa824137ce0 legacy=0x7fa82409d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 shutdown_connections 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 >> 172.21.15.78:0/3603946132 conn(0x7fa8240aa6c0 msgr2=0x7fa8240a2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 shutdown_connections 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 1 -- 172.21.15.78:0/3603946132 wait complete. 2024-09-15T20:37:16.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.656+0000 7fa82a8d6640 1 librados: shutdown 2024-09-15T20:37:16.671 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:37:16.671 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:37:16.671 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient: get_monmap_and_config 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient: build_initial_monmap 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient: monmap: 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:16.789548+0000 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:16.789548+0000 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:16.789 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:16.790 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 1 Processor -- start 2024-09-15T20:37:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 1 -- start start 2024-09-15T20:37:16.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient: init 2024-09-15T20:37:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:37:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient(hunting): picked mon.noname-a con 0x7fa08c137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient(hunting): picked mon.noname-c con 0x7fa08c074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient(hunting): picked mon.noname-b con 0x7fa08c13e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c138850 con 0x7fa08c137d60 2024-09-15T20:37:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c0a5da0 con 0x7fa08c074500 2024-09-15T20:37:16.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c073560 con 0x7fa08c13e370 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient(hunting): authenticate will time out at 1497.674915s 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08ad9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa08c13e370 0x7fa08c140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57198/0 (socket says 172.21.15.78:57198) 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08ad9d640 1 -- 172.21.15.78:0/3593585225 learned_addr learned my addr 172.21.15.78:0/3593585225 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08a59c640 1 --1- 172.21.15.78:0/3593585225 >> v1:172.21.15.78:6789/0 conn(0x7fa08c137d60 0x7fa08c138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50474/0 (socket says 172.21.15.78:50474) 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 823361517 0 0) 0x7fa08c138850 con 0x7fa08c137d60 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient(hunting): my global_id is 94098 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa074003530 con 0x7fa08c137d60 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 720805409 0 0) 0x7fa08c073560 con 0x7fa08c13e370 2024-09-15T20:37:16.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient(hunting): my global_id is 84314 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa08c138850 con 0x7fa08c13e370 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2278818747 0 0) 0x7fa08c138850 con 0x7fa08c13e370 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 >> v1:172.21.15.146:6789/0 conn(0x7fa08c074500 legacy=0x7fa08c13cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 >> v1:172.21.15.78:6789/0 conn(0x7fa08c137d60 legacy=0x7fa08c138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: _finish_hunting 0 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 monclient: found mon.noname-b 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c073840 con 0x7fa08c13e370 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: _finish_auth 0 2024-09-15T20:37:16.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: _check_auth_tickets 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa084001680 con 0x7fa08c13e370 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 5 monclient: authenticate success, global_id 84314 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: dump: 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.795 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa084001820 con 0x7fa08c13e370 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa090fc9640 4 set_mon_vals no callback set 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 1 -- 172.21.15.78:0/3593585225 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa084001c30 con 0x7fa08c13e370 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.788+0000 7fa08959a640 10 monclient: dump: 2024-09-15T20:37:16.796 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.797 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: shutdown 2024-09-15T20:37:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- 172.21.15.78:0/3593585225 >> v1:172.21.15.78:6790/0 conn(0x7fa08c13e370 legacy=0x7fa08c140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- 172.21.15.78:0/3593585225 shutdown_connections 2024-09-15T20:37:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- 172.21.15.78:0/3593585225 >> 172.21.15.78:0/3593585225 conn(0x7fa08c0aa650 msgr2=0x7fa08c0aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- 172.21.15.78:0/3593585225 shutdown_connections 2024-09-15T20:37:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- 172.21.15.78:0/3593585225 wait complete. 2024-09-15T20:37:16.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: build_initial_monmap 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: monmap: 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 librados: starting msgr at 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 librados: starting objecter 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 Processor -- start 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- start start 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 librados: setting wanted keys 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 librados: calling monclient init 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: init 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient(hunting): picked mon.noname-b con 0x7fa08c074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient(hunting): picked mon.noname-c con 0x7fa08c137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient(hunting): picked mon.noname-a con 0x7fa08c13e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c09e0b0 con 0x7fa08c13e370 2024-09-15T20:37:16.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c13a110 con 0x7fa08c074500 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa08c13bbd0 con 0x7fa08c137d60 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient(hunting): authenticate will time out at 1497.678127s 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa089d9b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa08c137d60 0x7fa08c1d1730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57202/0 (socket says 172.21.15.78:57202) 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa089d9b640 1 -- 172.21.15.78:0/2561865236 learned_addr learned my addr 172.21.15.78:0/2561865236 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 client.?.objecter ms_handle_connect 0x7fa08c137d60 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 client.?.objecter ms_handle_connect 0x7fa08c13e370 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 714057341 0 0) 0x7fa08c13bbd0 con 0x7fa08c137d60 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): my global_id is 84317 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa06c004ec0 con 0x7fa08c137d60 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340490255 0 0) 0x7fa08c09e0b0 con 0x7fa08c13e370 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): my global_id is 94101 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa08c13bbd0 con 0x7fa08c13e370 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 client.?.objecter ms_handle_connect 0x7fa08c074500 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2721314580 0 0) 0x7fa06c004ec0 con 0x7fa08c137d60 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa08c09e0b0 con 0x7fa08c137d60 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa080001a40 con 0x7fa08c137d60 2024-09-15T20:37:16.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): dump: 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:16.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2580012329 0 0) 0x7fa08c13bbd0 con 0x7fa08c13e370 2024-09-15T20:37:16.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa06c004ec0 con 0x7fa08c13e370 2024-09-15T20:37:16.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa084001b70 con 0x7fa08c13e370 2024-09-15T20:37:16.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:16.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:16.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): dump: 2024-09-15T20:37:16.808 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.808 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.808 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1951293665 0 0) 0x7fa08c13a110 con 0x7fa08c074500 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient(hunting): my global_id is 23377 2024-09-15T20:37:16.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa08c13bbd0 con 0x7fa08c074500 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2659515480 0 0) 0x7fa08c09e0b0 con 0x7fa08c137d60 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 >> v1:172.21.15.146:6789/0 conn(0x7fa08c074500 legacy=0x7fa08c139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 >> v1:172.21.15.78:6789/0 conn(0x7fa08c13e370 legacy=0x7fa08c1d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient: _finish_hunting 0 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 monclient: found mon.c 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa08c13c470 con 0x7fa08c137d60 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient: _finish_auth 0 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient: _check_auth_tickets 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 5 monclient: authenticate success, global_id 84317 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: _renew_subs 2024-09-15T20:37:16.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa08c138cb0 con 0x7fa08c137d60 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 client.84317.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: _renew_subs 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa08c13ae40 con 0x7fa08c137d60 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 librados: init done 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa080001d40 con 0x7fa08c137d60 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa080003a70 con 0x7fa08c137d60 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 10 monclient: dump: 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.811 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa088d99640 4 set_mon_vals no callback set 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa0700065d0 con 0x7fa070002870 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.792+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fa080003c10 con 0x7fa08c137d60 2024-09-15T20:37:16.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 client.84317.objecter ms_handle_connect 0x7fa070002870 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 client.84317.objecter resend_mon_ops 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(261..261 src has 1..261) ==== 5788+0+0 (unknown 2622532364 0 0) 0x7fa08004b9f0 con 0x7fa08c137d60 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 client.84317.objecter ms_dispatch 0x7fa08c000ba0 osd_map(261..261 src has 1..261) 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 3 client.84317.objecter handle_osd_map got epochs [261,261] > 0 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 3 client.84317.objecter handle_osd_map decoding full epoch 261 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 20 client.84317.objecter dump_active .. 0 homeless 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 440663149 0 0) 0x7fa0700065d0 con 0x7fa070002870 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient(hunting): my global_id is 94104 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa08c09e0b0 con 0x7fa070002870 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa08ad9d640 10 monclient: get_auth_request con 0x7fa06c03e1a0 auth_method 0 2024-09-15T20:37:16.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 254742454 0 0) 0x7fa08c09e0b0 con 0x7fa070002870 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa0700065d0 con 0x7fa070002870 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa07c001510 con 0x7fa070002870 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient: dump: 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.814 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2316399668 0 0) 0x7fa0700065d0 con 0x7fa070002870 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa07000a170 con 0x7fa070002870 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 client.84317.objecter ms_handle_connect 0x7fa06c03e1a0 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa07c004b70 con 0x7fa070002870 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 >> v1:172.21.15.78:6789/0 conn(0x7fa070002870 legacy=0x7fa070004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.796+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 >> v1:172.21.15.78:6789/0 conn(0x7fa070002870 legacy=0x7fa070004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa090fc9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa070009e80 con 0x7fa070009140 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 client.84317.objecter ms_handle_connect 0x7fa070009140 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 client.84317.objecter resend_mon_ops 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2970075727 0 0) 0x7fa070009e80 con 0x7fa070009140 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 monclient(hunting): my global_id is 94107 2024-09-15T20:37:16.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa0700065d0 con 0x7fa070009140 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 276030322 0 0) 0x7fa0700065d0 con 0x7fa070009140 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa070009e80 con 0x7fa070009140 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa07c004dc0 con 0x7fa070009140 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 10 monclient: dump: 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.818 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2156247950 0 0) 0x7fa070009e80 con 0x7fa070009140 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.800+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa07000c420 con 0x7fa070009140 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3970 (unknown 0 0 2836601757) 0x7fa07c005f90 con 0x7fa070009140 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa07affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa07affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 >> v1:172.21.15.78:6789/0 conn(0x7fa070009140 legacy=0x7fa0700095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa07affd640 1 -- 172.21.15.78:0/2561865236 >> v1:172.21.15.78:6789/0 conn(0x7fa070009140 legacy=0x7fa0700095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 24, 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 463, 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.820 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:16.821 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:16.822 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:37:16.823 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:16.824 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:16.825 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:16.826 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:16.829 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.830 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:37:16.831 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 10 librados: watch_flush enter 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 10 librados: watch_flush exit 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 20 client.84317.objecter shutdown clearing up homeless session... 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 10 client.84317.objecter successfully canceled tick 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 >> v1:172.21.15.146:6812/435598642 conn(0x7fa06c03e1a0 legacy=0x7fa06c040660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 10 monclient: shutdown 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 >> v1:172.21.15.78:6790/0 conn(0x7fa08c137d60 legacy=0x7fa08c1d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa08ad9d640 1 -- 172.21.15.78:0/2561865236 reap_dead start 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa08ad9d640 1 -- 172.21.15.78:0/2561865236 reap_dead start 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 shutdown_connections 2024-09-15T20:37:16.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 >> 172.21.15.78:0/2561865236 conn(0x7fa08c0aa650 msgr2=0x7fa08c0a2790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:16.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 shutdown_connections 2024-09-15T20:37:16.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 1 -- 172.21.15.78:0/2561865236 wait complete. 2024-09-15T20:37:16.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.804+0000 7fa090fc9640 1 librados: shutdown 2024-09-15T20:37:16.834 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:37:16.834 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:37:16.834 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:37:16.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: get_monmap_and_config 2024-09-15T20:37:16.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: build_initial_monmap 2024-09-15T20:37:16.944 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: monmap: 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:16.945056+0000 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:16.945056+0000 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:16.945 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 Processor -- start 2024-09-15T20:37:16.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- start start 2024-09-15T20:37:16.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: init 2024-09-15T20:37:16.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:16.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd4137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd40a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cd40a6420 con 0x7f2cd40a9610 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cd40a6700 con 0x7f2cd413e5f0 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cd40a2110 con 0x7f2cd4137ce0 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient(hunting): authenticate will time out at 1497.829390s 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cd8e09640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2cd40a9610 0x7f2cd413ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50522/0 (socket says 172.21.15.78:50522) 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cd8e09640 1 -- 172.21.15.78:0/938469605 learned_addr learned my addr 172.21.15.78:0/938469605 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3760933075 0 0) 0x7f2cd40a6420 con 0x7f2cd40a9610 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient(hunting): my global_id is 94110 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cc0003530 con 0x7f2cd40a9610 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3559496514 0 0) 0x7f2cd40a2110 con 0x7f2cd4137ce0 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient(hunting): my global_id is 84320 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cd40a6420 con 0x7f2cd4137ce0 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4287481678 0 0) 0x7f2cc0003530 con 0x7f2cd40a9610 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 >> v1:172.21.15.78:6790/0 conn(0x7f2cd4137ce0 legacy=0x7f2cd41380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 >> v1:172.21.15.146:6789/0 conn(0x7f2cd413e5f0 legacy=0x7f2cd41409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: _finish_hunting 0 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 monclient: found mon.noname-a 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd40a4bc0 con 0x7f2cd40a9610 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: _finish_auth 0 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: _check_auth_tickets 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2cc40016a0 con 0x7f2cd40a9610 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 5 monclient: authenticate success, global_id 94110 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: dump: 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:16.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2cc40018c0 con 0x7f2cd40a9610 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 4 set_mon_vals no callback set 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 1 -- 172.21.15.78:0/938469605 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2cc4001cd0 con 0x7f2cd40a9610 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2ccbfff640 10 monclient: dump: 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.959 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: shutdown 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- 172.21.15.78:0/938469605 >> v1:172.21.15.78:6789/0 conn(0x7f2cd40a9610 legacy=0x7f2cd413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- 172.21.15.78:0/938469605 shutdown_connections 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- 172.21.15.78:0/938469605 >> 172.21.15.78:0/938469605 conn(0x7f2cd40aa6c0 msgr2=0x7f2cd40aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- 172.21.15.78:0/938469605 shutdown_connections 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 -- 172.21.15.78:0/938469605 wait complete. 2024-09-15T20:37:16.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: build_initial_monmap 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 10 monclient: monmap: 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 librados: starting msgr at 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 librados: starting objecter 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.944+0000 7f2cdb86e640 1 Processor -- start 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 -- start start 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 librados: setting wanted keys 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 librados: calling monclient init 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient: init 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient(hunting): picked mon.noname-a con 0x7f2cd40a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient(hunting): picked mon.noname-c con 0x7f2cd4137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient(hunting): picked mon.noname-b con 0x7f2cd413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cd4074330 con 0x7f2cd40a9610 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cd40a14b0 con 0x7f2cd413e5f0 2024-09-15T20:37:16.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cd413be00 con 0x7f2cd4137ce0 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient(hunting): authenticate will time out at 1497.832470s 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cd8e09640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f2cd4137ce0 0x7f2cd4139a80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57218/0 (socket says 172.21.15.78:57218) 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cd8e09640 1 -- 172.21.15.78:0/3603039607 learned_addr learned my addr 172.21.15.78:0/3603039607 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.?.objecter ms_handle_connect 0x7f2cd4137ce0 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.?.objecter ms_handle_connect 0x7f2cd40a9610 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1132221338 0 0) 0x7f2cd4074330 con 0x7f2cd40a9610 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): my global_id is 94113 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cb8004ec0 con 0x7f2cd40a9610 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1427401452 0 0) 0x7f2cd413be00 con 0x7f2cd4137ce0 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): my global_id is 84323 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cd4074330 con 0x7f2cd4137ce0 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.?.objecter ms_handle_connect 0x7f2cd413e5f0 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:16.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1093283429 0 0) 0x7f2cd4074330 con 0x7f2cd4137ce0 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2cd413be00 con 0x7f2cd4137ce0 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2cc4001c10 con 0x7f2cd4137ce0 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): dump: 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.965 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3575814670 0 0) 0x7f2cb8004ec0 con 0x7f2cd40a9610 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2cd4074330 con 0x7f2cd40a9610 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2ccc001cc0 con 0x7f2cd40a9610 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): dump: 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.966 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3047712792 0 0) 0x7f2cd40a14b0 con 0x7f2cd413e5f0 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): my global_id is 23383 2024-09-15T20:37:16.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cb8004ec0 con 0x7f2cd413e5f0 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1535137074 0 0) 0x7f2cd413be00 con 0x7f2cd4137ce0 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 >> v1:172.21.15.146:6789/0 conn(0x7f2cd413e5f0 legacy=0x7f2cd41d48f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 >> v1:172.21.15.78:6789/0 conn(0x7f2cd40a9610 legacy=0x7f2cd4139370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 monclient: found mon.c 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd413c0e0 con 0x7f2cd4137ce0 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:16.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 5 monclient: authenticate success, global_id 84323 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient: _renew_subs 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd413abb0 con 0x7f2cd4137ce0 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 client.84323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient: _renew_subs 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f2cd41d6010 con 0x7f2cd4137ce0 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 librados: init done 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2cc4001820 con 0x7f2cd4137ce0 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2cc4004da0 con 0x7f2cd4137ce0 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: dump: 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2ccb7fe640 4 set_mon_vals no callback set 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2cb40065d0 con 0x7f2cb4002870 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.84323.objecter ms_handle_connect 0x7f2cb4002870 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.84323.objecter resend_mon_ops 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f2cc4010e10 con 0x7f2cd4137ce0 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(261..261 src has 1..261) ==== 5788+0+0 (unknown 2622532364 0 0) 0x7f2cc404b970 con 0x7f2cd4137ce0 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.84323.objecter ms_dispatch 0x7f2cd4000ba0 osd_map(261..261 src has 1..261) 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 3 client.84323.objecter handle_osd_map got epochs [261,261] > 0 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 3 client.84323.objecter handle_osd_map decoding full epoch 261 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 20 client.84323.objecter dump_active .. 0 homeless 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2071063640 0 0) 0x7f2cb40065d0 con 0x7f2cb4002870 2024-09-15T20:37:16.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient(hunting): my global_id is 84326 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cd413be00 con 0x7f2cb4002870 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cd9e0b640 10 monclient: get_auth_request con 0x7f2cb803e240 auth_method 0 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1521215744 0 0) 0x7f2cd413be00 con 0x7f2cb4002870 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2cb40065d0 con 0x7f2cb4002870 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2ccc003b20 con 0x7f2cb4002870 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: dump: 2024-09-15T20:37:16.972 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3387514710 0 0) 0x7f2cb40065d0 con 0x7f2cb4002870 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cb400a170 con 0x7f2cb4002870 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 client.84323.objecter ms_handle_connect 0x7f2cb803e240 2024-09-15T20:37:16.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2ccc005060 con 0x7f2cb4002870 2024-09-15T20:37:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 >> v1:172.21.15.78:6790/0 conn(0x7f2cb4002870 legacy=0x7f2cb4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.948+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 >> v1:172.21.15.78:6790/0 conn(0x7f2cb4002870 legacy=0x7f2cb4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:37:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2cb4009e10 con 0x7f2cb40090f0 2024-09-15T20:37:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 client.84323.objecter ms_handle_connect 0x7f2cb40090f0 2024-09-15T20:37:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 client.84323.objecter resend_mon_ops 2024-09-15T20:37:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2932456656 0 0) 0x7f2cb4009e10 con 0x7f2cb40090f0 2024-09-15T20:37:16.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient(hunting): my global_id is 84329 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2cb40065d0 con 0x7f2cb40090f0 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3953947113 0 0) 0x7f2cb40065d0 con 0x7f2cb40090f0 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2cb4009e10 con 0x7f2cb40090f0 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2ccc005800 con 0x7f2cb40090f0 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient: dump: 2024-09-15T20:37:16.976 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 369998636 0 0) 0x7f2cb4009e10 con 0x7f2cb40090f0 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2cb400c3b0 con 0x7f2cb40090f0 2024-09-15T20:37:16.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 1620842912) 0x7f2ccc0058c0 con 0x7f2cb40090f0 2024-09-15T20:37:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:37:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 >> v1:172.21.15.78:6790/0 conn(0x7f2cb40090f0 legacy=0x7f2cb4009560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cc9ffb640 1 -- 172.21.15.78:0/3603039607 >> v1:172.21.15.78:6790/0 conn(0x7f2cb40090f0 legacy=0x7f2cb4009560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:16.978 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:37:16.978 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:16.978 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:16.978 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 24, 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 108642, 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:16.979 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:37:16.980 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:37:16.982 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:16.983 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:16.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.985 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:16.985 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:37:16.985 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:16.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.988 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.988 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:37:16.988 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:37:16.988 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:16.988 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:16.988 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:16.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:37:16.989 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:37:16.990 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:37:16.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.992 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:37:16.993 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:37:16.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 10 librados: watch_flush enter 2024-09-15T20:37:16.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 10 librados: watch_flush exit 2024-09-15T20:37:16.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 20 client.84323.objecter shutdown clearing up homeless session... 2024-09-15T20:37:16.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 10 client.84323.objecter successfully canceled tick 2024-09-15T20:37:16.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 >> v1:172.21.15.146:6812/435598642 conn(0x7f2cb803e240 legacy=0x7f2cb8040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 10 monclient: shutdown 2024-09-15T20:37:16.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:16.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 >> v1:172.21.15.78:6790/0 conn(0x7f2cd4137ce0 legacy=0x7f2cd4139a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:16.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cd9e0b640 1 -- 172.21.15.78:0/3603039607 reap_dead start 2024-09-15T20:37:16.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cd9e0b640 1 -- 172.21.15.78:0/3603039607 reap_dead start 2024-09-15T20:37:16.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 shutdown_connections 2024-09-15T20:37:16.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 >> 172.21.15.78:0/3603039607 conn(0x7f2cd40aa6c0 msgr2=0x7f2cd41409e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:16.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 shutdown_connections 2024-09-15T20:37:16.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 1 -- 172.21.15.78:0/3603039607 wait complete. 2024-09-15T20:37:16.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:16.956+0000 7f2cdb86e640 1 librados: shutdown 2024-09-15T20:37:16.996 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:37:16.997 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:37:16.997 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:37:17.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: get_monmap_and_config 2024-09-15T20:37:17.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: build_initial_monmap 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: monmap: 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:17.117073+0000 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:17.117073+0000 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:17.117 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 Processor -- start 2024-09-15T20:37:17.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- start start 2024-09-15T20:37:17.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: init 2024-09-15T20:37:17.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:17.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:37:17.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient(hunting): picked mon.noname-c con 0x7fb578137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:17.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient(hunting): picked mon.noname-b con 0x7fb5780a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:17.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient(hunting): picked mon.noname-a con 0x7fb57813e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:17.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5780a6420 con 0x7fb57813e5f0 2024-09-15T20:37:17.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5780a6700 con 0x7fb578137ce0 2024-09-15T20:37:17.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5780a2110 con 0x7fb5780a9610 2024-09-15T20:37:17.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient(hunting): authenticate will time out at 1498.001709s 2024-09-15T20:37:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57c828640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb57813e5f0 0x7fb5781409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50536/0 (socket says 172.21.15.78:50536) 2024-09-15T20:37:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57c828640 1 -- 172.21.15.78:0/1999372763 learned_addr learned my addr 172.21.15.78:0/1999372763 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb5777fe640 1 --1- 172.21.15.78:0/1999372763 >> v1:172.21.15.78:6790/0 conn(0x7fb5780a9610 0x7fb57813ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57246/0 (socket says 172.21.15.78:57246) 2024-09-15T20:37:17.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57c828640 1 -- 172.21.15.78:0/1999372763 >> v1:172.21.15.78:6789/0 conn(0x7fb57813e5f0 legacy=0x7fb5781409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 15 2024-09-15T20:37:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57c828640 1 -- 172.21.15.78:0/1999372763 >> v1:172.21.15.78:6789/0 conn(0x7fb57813e5f0 legacy=0x7fb5781409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T20:37:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57c828640 1 --1- 172.21.15.78:0/1999372763 >> v1:172.21.15.78:6789/0 conn(0x7fb57813e5f0 0x7fb5781409e0 :-1 s=OPENED pgs=8966 cs=1 l=1).handle_message read tag failed 2024-09-15T20:37:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57c828640 1 --1- 172.21.15.78:0/1999372763 >> v1:172.21.15.78:6789/0 conn(0x7fb57813e5f0 0x7fb5781409e0 :-1 s=OPENED pgs=8966 cs=1 l=1).fault on lossy channel, failing 2024-09-15T20:37:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.78:6789/0 2024-09-15T20:37:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2272911389 0 0) 0x7fb5780a2110 con 0x7fb5780a9610 2024-09-15T20:37:17.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient(hunting): my global_id is 84332 2024-09-15T20:37:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb560003530 con 0x7fb5780a9610 2024-09-15T20:37:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3360552989 0 0) 0x7fb560003530 con 0x7fb5780a9610 2024-09-15T20:37:17.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 >> v1:172.21.15.146:6789/0 conn(0x7fb578137ce0 legacy=0x7fb5781380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 >> v1:172.21.15.78:6789/0 conn(0x7fb57813e5f0 legacy=0x7fb5781409e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:37:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 monclient: found mon.noname-b 2024-09-15T20:37:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5780a4bc0 con 0x7fb5780a9610 2024-09-15T20:37:17.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: _finish_auth 0 2024-09-15T20:37:17.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:37:17.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:17.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb56c0016a0 con 0x7fb5780a9610 2024-09-15T20:37:17.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 5 monclient: authenticate success, global_id 84332 2024-09-15T20:37:17.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:17.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:17.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:17.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: dump: 2024-09-15T20:37:17.134 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.134 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.135 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.135 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.135 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.135 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.135 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.136 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.136 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.136 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:17.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb56c0018c0 con 0x7fb5780a9610 2024-09-15T20:37:17.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:17.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 1 -- 172.21.15.78:0/1999372763 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb56c001cd0 con 0x7fb5780a9610 2024-09-15T20:37:17.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:17.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:17.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb576ffd640 10 monclient: dump: 2024-09-15T20:37:17.137 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.137 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.138 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.138 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.138 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.138 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.138 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.139 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.139 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.139 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:17.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 4 set_mon_vals no callback set 2024-09-15T20:37:17.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: shutdown 2024-09-15T20:37:17.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:17.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- 172.21.15.78:0/1999372763 >> v1:172.21.15.78:6790/0 conn(0x7fb5780a9610 legacy=0x7fb57813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- 172.21.15.78:0/1999372763 shutdown_connections 2024-09-15T20:37:17.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- 172.21.15.78:0/1999372763 >> 172.21.15.78:0/1999372763 conn(0x7fb5780aa6c0 msgr2=0x7fb5780aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:17.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- 172.21.15.78:0/1999372763 shutdown_connections 2024-09-15T20:37:17.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 1 -- 172.21.15.78:0/1999372763 wait complete. 2024-09-15T20:37:17.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: build_initial_monmap 2024-09-15T20:37:17.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.116+0000 7fb57e28b640 10 monclient: monmap: 2024-09-15T20:37:17.141 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:17.141 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:17.141 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:17.142 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:17.142 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:17.142 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.142 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:17.142 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:17.142 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:17.143 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 librados: starting msgr at 2024-09-15T20:37:17.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 librados: starting objecter 2024-09-15T20:37:17.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 Processor -- start 2024-09-15T20:37:17.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 -- start start 2024-09-15T20:37:17.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 librados: setting wanted keys 2024-09-15T20:37:17.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 librados: calling monclient init 2024-09-15T20:37:17.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient: init 2024-09-15T20:37:17.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:17.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:37:17.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient(hunting): picked mon.noname-b con 0x7fb5780a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:17.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient(hunting): picked mon.noname-c con 0x7fb578137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:17.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient(hunting): picked mon.noname-a con 0x7fb57813e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:17.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5781cddb0 con 0x7fb57813e5f0 2024-09-15T20:37:17.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb578138fe0 con 0x7fb5780a9610 2024-09-15T20:37:17.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb57813e2c0 con 0x7fb578137ce0 2024-09-15T20:37:17.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:17.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient(hunting): authenticate will time out at 1498.004852s 2024-09-15T20:37:17.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb5777fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb578137ce0 0x7fb5781cd6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57256/0 (socket says 172.21.15.78:57256) 2024-09-15T20:37:17.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb5777fe640 1 -- 172.21.15.78:0/421152253 learned_addr learned my addr 172.21.15.78:0/421152253 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:17.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.?.objecter ms_handle_connect 0x7fb578137ce0 2024-09-15T20:37:17.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:17.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.?.objecter ms_handle_connect 0x7fb57813e5f0 2024-09-15T20:37:17.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:17.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2776579029 0 0) 0x7fb57813e2c0 con 0x7fb578137ce0 2024-09-15T20:37:17.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): my global_id is 84335 2024-09-15T20:37:17.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb564004ec0 con 0x7fb578137ce0 2024-09-15T20:37:17.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 996214130 0 0) 0x7fb5781cddb0 con 0x7fb57813e5f0 2024-09-15T20:37:17.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): my global_id is 94116 2024-09-15T20:37:17.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb57813e2c0 con 0x7fb57813e5f0 2024-09-15T20:37:17.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.?.objecter ms_handle_connect 0x7fb5780a9610 2024-09-15T20:37:17.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:17.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1265739295 0 0) 0x7fb564004ec0 con 0x7fb578137ce0 2024-09-15T20:37:17.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5781cddb0 con 0x7fb578137ce0 2024-09-15T20:37:17.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb56c001c10 con 0x7fb578137ce0 2024-09-15T20:37:17.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:17.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:17.150 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): dump: 2024-09-15T20:37:17.150 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.151 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.151 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.151 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.151 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.151 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.151 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.152 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.152 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.152 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:17.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 909531890 0 0) 0x7fb57813e2c0 con 0x7fb57813e5f0 2024-09-15T20:37:17.152 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb564004ec0 con 0x7fb57813e5f0 2024-09-15T20:37:17.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb5700018b0 con 0x7fb57813e5f0 2024-09-15T20:37:17.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:17.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:17.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): dump: 2024-09-15T20:37:17.153 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.154 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.154 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.154 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.154 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.155 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.155 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.155 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.155 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.155 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.155 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 221056017 0 0) 0x7fb578138fe0 con 0x7fb5780a9610 2024-09-15T20:37:17.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): my global_id is 23389 2024-09-15T20:37:17.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb57813e2c0 con 0x7fb5780a9610 2024-09-15T20:37:17.156 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 591527216 0 0) 0x7fb5781cddb0 con 0x7fb578137ce0 2024-09-15T20:37:17.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 >> v1:172.21.15.146:6789/0 conn(0x7fb5780a9610 legacy=0x7fb5781ccf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 >> v1:172.21.15.78:6789/0 conn(0x7fb57813e5f0 legacy=0x7fb5781d51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:37:17.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 monclient: found mon.c 2024-09-15T20:37:17.157 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:17.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb57809f480 con 0x7fb578137ce0 2024-09-15T20:37:17.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: _finish_auth 0 2024-09-15T20:37:17.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:37:17.158 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 5 monclient: authenticate success, global_id 84335 2024-09-15T20:37:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient: _renew_subs 2024-09-15T20:37:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb5780a05b0 con 0x7fb578137ce0 2024-09-15T20:37:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 client.84335.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:17.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient: _renew_subs 2024-09-15T20:37:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb57809dea0 con 0x7fb578137ce0 2024-09-15T20:37:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 librados: init done 2024-09-15T20:37:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb56c001820 con 0x7fb578137ce0 2024-09-15T20:37:17.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:17.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb56c004aa0 con 0x7fb578137ce0 2024-09-15T20:37:17.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:17.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:17.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb5767fc640 4 set_mon_vals no callback set 2024-09-15T20:37:17.161 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: dump: 2024-09-15T20:37:17.161 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.162 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.162 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.162 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.162 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.162 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.162 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.163 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.163 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.163 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:17.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb5580065d0 con 0x7fb558002870 2024-09-15T20:37:17.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb56c010b10 con 0x7fb578137ce0 2024-09-15T20:37:17.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.84335.objecter ms_handle_connect 0x7fb558002870 2024-09-15T20:37:17.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.84335.objecter resend_mon_ops 2024-09-15T20:37:17.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(261..261 src has 1..261) ==== 5788+0+0 (unknown 2622532364 0 0) 0x7fb56c04b680 con 0x7fb578137ce0 2024-09-15T20:37:17.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.84335.objecter ms_dispatch 0x7fb578000ba0 osd_map(261..261 src has 1..261) 2024-09-15T20:37:17.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 3 client.84335.objecter handle_osd_map got epochs [261,261] > 0 2024-09-15T20:37:17.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 3 client.84335.objecter handle_osd_map decoding full epoch 261 2024-09-15T20:37:17.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 20 client.84335.objecter dump_active .. 0 homeless 2024-09-15T20:37:17.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 173492314 0 0) 0x7fb5580065d0 con 0x7fb558002870 2024-09-15T20:37:17.164 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient(hunting): my global_id is 23392 2024-09-15T20:37:17.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5781cddb0 con 0x7fb558002870 2024-09-15T20:37:17.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb57c828640 10 monclient: get_auth_request con 0x7fb56403e220 auth_method 0 2024-09-15T20:37:17.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1438987834 0 0) 0x7fb5781cddb0 con 0x7fb558002870 2024-09-15T20:37:17.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb5580065d0 con 0x7fb558002870 2024-09-15T20:37:17.165 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb568001b10 con 0x7fb558002870 2024-09-15T20:37:17.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:17.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:37:17.166 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 monclient: dump: 2024-09-15T20:37:17.166 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.166 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.167 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.167 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.167 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.167 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.167 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.167 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.167 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.168 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.120+0000 7fb574ff9640 10 client.84335.objecter ms_handle_connect 0x7fb56403e220 2024-09-15T20:37:17.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.124+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 668030523 0 0) 0x7fb5580065d0 con 0x7fb558002870 2024-09-15T20:37:17.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.124+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb55800a170 con 0x7fb558002870 2024-09-15T20:37:17.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.124+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb568001db0 con 0x7fb558002870 2024-09-15T20:37:17.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.124+0000 7fb574ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:17.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.124+0000 7fb574ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:17.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.124+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 >> v1:172.21.15.146:6789/0 conn(0x7fb558002870 legacy=0x7fb558004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.124+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 >> v1:172.21.15.146:6789/0 conn(0x7fb558002870 legacy=0x7fb558004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb57e28b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb558009e60 con 0x7fb558009120 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb574ff9640 10 client.84335.objecter ms_handle_connect 0x7fb558009120 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb574ff9640 10 client.84335.objecter resend_mon_ops 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3645038187 0 0) 0x7fb558009e60 con 0x7fb558009120 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb574ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb574ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb574ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb574ff9640 10 monclient(hunting): my global_id is 23395 2024-09-15T20:37:17.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.128+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5580065d0 con 0x7fb558009120 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1738920106 0 0) 0x7fb5580065d0 con 0x7fb558009120 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb558009e60 con 0x7fb558009120 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb568003b20 con 0x7fb558009120 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 10 monclient: dump: 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.172 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1240527394 0 0) 0x7fb558009e60 con 0x7fb558009120 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb55800c3b0 con 0x7fb558009120 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 1022700991) 0x7fb5680049d0 con 0x7fb558009120 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:17.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 >> v1:172.21.15.146:6789/0 conn(0x7fb558009120 legacy=0x7fb5580095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb574ff9640 1 -- 172.21.15.78:0/421152253 >> v1:172.21.15.146:6789/0 conn(0x7fb558009120 legacy=0x7fb5580095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:17.174 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:37:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:37:17.174 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 24, 2024-09-15T20:37:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 349640, 2024-09-15T20:37:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:37:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:37:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:37:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:37:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:17.175 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:17.176 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:37:17.177 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:17.178 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:17.179 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:17.179 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:17.179 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.179 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:37:17.179 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:37:17.179 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:37:17.179 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:37:17.180 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:37:17.180 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:37:17.180 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:17.180 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:17.180 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:37:17.180 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:37:17.180 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:37:17.180 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:17.181 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:17.182 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:17.183 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.184 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.185 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:17.186 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.187 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:37:17.188 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:17.189 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:17.189 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:17.189 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:17.189 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:17.189 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:17.189 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:17.189 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:37:17.189 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 10 librados: watch_flush enter 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 10 librados: watch_flush exit 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 20 client.84335.objecter shutdown clearing up homeless session... 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 10 client.84335.objecter successfully canceled tick 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 >> v1:172.21.15.146:6812/435598642 conn(0x7fb56403e220 legacy=0x7fb5640406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 10 monclient: shutdown 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 >> v1:172.21.15.78:6790/0 conn(0x7fb578137ce0 legacy=0x7fb5781cd6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57c828640 1 -- 172.21.15.78:0/421152253 reap_dead start 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57c828640 1 -- 172.21.15.78:0/421152253 reap_dead start 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 shutdown_connections 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 >> 172.21.15.78:0/421152253 conn(0x7fb5780aa6c0 msgr2=0x7fb5780a2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:17.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 shutdown_connections 2024-09-15T20:37:17.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 1 -- 172.21.15.78:0/421152253 wait complete. 2024-09-15T20:37:17.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.132+0000 7fb57e28b640 1 librados: shutdown 2024-09-15T20:37:17.192 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:37:17.192 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:37:17.192 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:37:17.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.304+0000 7f1048492640 10 monclient: get_monmap_and_config 2024-09-15T20:37:17.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.304+0000 7f1048492640 10 monclient: build_initial_monmap 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.304+0000 7f1048492640 10 monclient: monmap: 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:17.308127+0000 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:17.308127+0000 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:17.308 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 Processor -- start 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- start start 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient: init 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient(hunting): picked mon.noname-b con 0x7f1040137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient(hunting): picked mon.noname-c con 0x7f1040074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient(hunting): picked mon.noname-a con 0x7f104013e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:17.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1040138850 con 0x7f104013e370 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10400a5da0 con 0x7f1040074500 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1040073560 con 0x7f1040137d60 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient(hunting): authenticate will time out at 1498.193141s 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104622e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1040137d60 0x7f1040138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57272/0 (socket says 172.21.15.78:57272) 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1046a2f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f104013e370 0x7f1040140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50556/0 (socket says 172.21.15.78:50556) 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104622e640 1 -- 172.21.15.78:0/169547024 learned_addr learned my addr 172.21.15.78:0/169547024 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2082449308 0 0) 0x7f1040073560 con 0x7f1040137d60 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient(hunting): my global_id is 84338 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f102c003530 con 0x7f1040137d60 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524398056 0 0) 0x7f1040138850 con 0x7f104013e370 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient(hunting): my global_id is 94119 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1040073560 con 0x7f104013e370 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 915758238 0 0) 0x7f102c003530 con 0x7f1040137d60 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 >> v1:172.21.15.146:6789/0 conn(0x7f1040074500 legacy=0x7f104013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 >> v1:172.21.15.78:6789/0 conn(0x7f104013e370 legacy=0x7f1040140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: _finish_hunting 0 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 monclient: found mon.noname-b 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1040073840 con 0x7f1040137d60 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: _finish_auth 0 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: _check_auth_tickets 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f10340015d0 con 0x7f1040137d60 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 5 monclient: authenticate success, global_id 84338 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:17.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: dump: 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1034001770 con 0x7f1040137d60 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 1 -- 172.21.15.78:0/169547024 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1034001b80 con 0x7f1040137d60 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 4 set_mon_vals no callback set 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f104522c640 10 monclient: dump: 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.321 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient: shutdown 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- 172.21.15.78:0/169547024 >> v1:172.21.15.78:6790/0 conn(0x7f1040137d60 legacy=0x7f1040138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- 172.21.15.78:0/169547024 shutdown_connections 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- 172.21.15.78:0/169547024 >> 172.21.15.78:0/169547024 conn(0x7f10400aa650 msgr2=0x7f10400aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:17.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- 172.21.15.78:0/169547024 shutdown_connections 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- 172.21.15.78:0/169547024 wait complete. 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient: build_initial_monmap 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient: monmap: 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:17.323 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 librados: starting msgr at 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 librados: starting objecter 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 Processor -- start 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 -- start start 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 librados: setting wanted keys 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 1 librados: calling monclient init 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.308+0000 7f1048492640 10 monclient: init 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:37:17.324 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient(hunting): picked mon.noname-a con 0x7f1040074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient(hunting): picked mon.noname-b con 0x7f1040137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient(hunting): picked mon.noname-c con 0x7f104013e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10401419f0 con 0x7f1040074500 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1040136a70 con 0x7f1040137d60 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10401325d0 con 0x7f104013e370 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient(hunting): authenticate will time out at 1498.196402s 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f104622e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1040074500 0x7f104009e6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50568/0 (socket says 172.21.15.78:50568) 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f104622e640 1 -- 172.21.15.78:0/1876599728 learned_addr learned my addr 172.21.15.78:0/1876599728 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 client.?.objecter ms_handle_connect 0x7f1040074500 2024-09-15T20:37:17.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 client.?.objecter ms_handle_connect 0x7f104013e370 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3099449347 0 0) 0x7f10401325d0 con 0x7f104013e370 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): my global_id is 84341 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1024004ec0 con 0x7f104013e370 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 560995006 0 0) 0x7f10401419f0 con 0x7f1040074500 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): my global_id is 94122 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10401325d0 con 0x7f1040074500 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 client.?.objecter ms_handle_connect 0x7f1040137d60 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2462421602 0 0) 0x7f1024004ec0 con 0x7f104013e370 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f10401419f0 con 0x7f104013e370 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1038001220 con 0x7f104013e370 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): dump: 2024-09-15T20:37:17.327 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2416572709 0 0) 0x7f10401325d0 con 0x7f1040074500 2024-09-15T20:37:17.328 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1024004ec0 con 0x7f1040074500 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1034001ac0 con 0x7f1040074500 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): dump: 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.329 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3308607584 0 0) 0x7f1040136a70 con 0x7f1040137d60 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient(hunting): my global_id is 23401 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f10401325d0 con 0x7f1040137d60 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3721272323 0 0) 0x7f10401419f0 con 0x7f104013e370 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 >> v1:172.21.15.146:6789/0 conn(0x7f1040137d60 legacy=0x7f104009edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 >> v1:172.21.15.78:6789/0 conn(0x7f1040074500 legacy=0x7f104009e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 monclient: found mon.c 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10401328b0 con 0x7f104013e370 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient: _finish_auth 0 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 5 monclient: authenticate success, global_id 84341 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient: _renew_subs 2024-09-15T20:37:17.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f10400a0030 con 0x7f104013e370 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 client.84341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient: _renew_subs 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f10400a0d20 con 0x7f104013e370 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 1 librados: init done 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f10380047f0 con 0x7f104013e370 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1038004b40 con 0x7f104013e370 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:17.332 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1044a2b640 4 set_mon_vals no callback set 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 monclient: dump: 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:17.333 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1038010bb0 con 0x7f104013e370 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1028005090 con 0x7f104013e370 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(261..261 src has 1..261) ==== 5788+0+0 (unknown 2622532364 0 0) 0x7f103804b7e0 con 0x7f104013e370 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 client.84341.objecter ms_dispatch 0x7f1040000ba0 osd_map(261..261 src has 1..261) 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 3 client.84341.objecter handle_osd_map got epochs [261,261] > 0 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 3 client.84341.objecter handle_osd_map decoding full epoch 261 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 20 client.84341.objecter dump_active .. 0 homeless 2024-09-15T20:37:17.334 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f104622e640 10 monclient: get_auth_request con 0x7f102403e300 auth_method 0 2024-09-15T20:37:17.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.312+0000 7f1032ffd640 10 client.84341.objecter ms_handle_connect 0x7f102403e300 2024-09-15T20:37:17.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.316+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f10380168d0 con 0x7f104013e370 2024-09-15T20:37:17.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.316+0000 7f1032ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:17.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.316+0000 7f1032ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.556+0000 7f1048492640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:37:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.556+0000 7f1048492640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:37:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.556+0000 7f1048492640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:17.557 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.556+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1028005c50 con 0x7f104013e370 2024-09-15T20:37:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.556+0000 7f1032ffd640 1 -- 172.21.15.78:0/1876599728 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f103801ac50 con 0x7f104013e370 2024-09-15T20:37:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.556+0000 7f1032ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:37:17.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.556+0000 7f1032ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:17.560 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.560 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=4018396759,config=2586702516,health=534850548,kv=847209388,logm=2239965294}) 2024-09-15T20:37:17.560 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=12,kv=9,logm=62} crc {auth=3922287462,config=2586702516,health=534850548,kv=2432466411,logm=1208945665}) 2024-09-15T20:37:17.560 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.560 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=4018396759,config=2586702516,health=534850548,kv=847209388,logm=2239965294}) 2024-09-15T20:37:17.560 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=15,config=2,health=12,kv=11,logm=60} crc {auth=1393856555,config=2586702516,health=534850548,kv=3449721867,logm=104354735}) 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 10 librados: watch_flush enter 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 10 librados: watch_flush exit 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 20 client.84341.objecter shutdown clearing up homeless session... 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 10 client.84341.objecter successfully canceled tick 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 >> v1:172.21.15.146:6812/435598642 conn(0x7f102403e300 legacy=0x7f10240407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 10 monclient: shutdown 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 >> v1:172.21.15.78:6790/0 conn(0x7f104013e370 legacy=0x7f104009f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 shutdown_connections 2024-09-15T20:37:17.562 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 >> 172.21.15.78:0/1876599728 conn(0x7f10400aa650 msgr2=0x7f10400a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 shutdown_connections 2024-09-15T20:37:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 1 -- 172.21.15.78:0/1876599728 wait complete. 2024-09-15T20:37:17.563 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f1048492640 1 librados: shutdown 2024-09-15T20:37:17.563 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.563 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3065948428}) 2024-09-15T20:37:17.563 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1876022997}) 2024-09-15T20:37:17.564 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.564 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3065948428}) 2024-09-15T20:37:17.564 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.560+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1331740613}) 2024-09-15T20:37:17.565 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.565 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568702730}) 2024-09-15T20:37:17.565 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=672072189}) 2024-09-15T20:37:17.566 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.566 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2568702730}) 2024-09-15T20:37:17.566 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3456714629}) 2024-09-15T20:37:17.566 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.567 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395646039}) 2024-09-15T20:37:17.567 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3757901434}) 2024-09-15T20:37:17.567 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.567 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2395646039}) 2024-09-15T20:37:17.567 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.564+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3864567598}) 2024-09-15T20:37:17.568 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.568 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999803348}) 2024-09-15T20:37:17.568 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2653876206}) 2024-09-15T20:37:17.568 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.569 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1999803348}) 2024-09-15T20:37:17.569 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3390431687}) 2024-09-15T20:37:17.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3293388812}) 2024-09-15T20:37:17.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=839167786}) 2024-09-15T20:37:17.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3293388812}) 2024-09-15T20:37:17.570 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1713139652}) 2024-09-15T20:37:17.571 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021298753}) 2024-09-15T20:37:17.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=590089291}) 2024-09-15T20:37:17.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2021298753}) 2024-09-15T20:37:17.572 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.568+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3689184946}) 2024-09-15T20:37:17.573 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.573 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4095192628}) 2024-09-15T20:37:17.573 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=386000149}) 2024-09-15T20:37:17.573 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.573 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4095192628}) 2024-09-15T20:37:17.574 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=791837571}) 2024-09-15T20:37:17.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853253508}) 2024-09-15T20:37:17.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4245765992}) 2024-09-15T20:37:17.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=853253508}) 2024-09-15T20:37:17.575 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.572+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2658454254}) 2024-09-15T20:37:17.576 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.576 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219956516}) 2024-09-15T20:37:17.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3818567512}) 2024-09-15T20:37:17.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3219956516}) 2024-09-15T20:37:17.577 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2496531477}) 2024-09-15T20:37:17.578 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:37:17.578 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801893398}) 2024-09-15T20:37:17.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=220687890}) 2024-09-15T20:37:17.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3801893398}) 2024-09-15T20:37:17.579 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2199657575}) 2024-09-15T20:37:17.580 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.580 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715537005}) 2024-09-15T20:37:17.580 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1704238480}) 2024-09-15T20:37:17.580 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.580 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3715537005}) 2024-09-15T20:37:17.580 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.576+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1032316937}) 2024-09-15T20:37:17.581 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.582 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516838406}) 2024-09-15T20:37:17.582 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1368301704}) 2024-09-15T20:37:17.582 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.582 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2516838406}) 2024-09-15T20:37:17.582 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1770779228}) 2024-09-15T20:37:17.583 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.583 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2192191112}) 2024-09-15T20:37:17.583 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2958318992}) 2024-09-15T20:37:17.583 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.583 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2192191112}) 2024-09-15T20:37:17.583 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.580+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1407817566}) 2024-09-15T20:37:17.584 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.585 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186792967}) 2024-09-15T20:37:17.585 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1579211337}) 2024-09-15T20:37:17.585 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.585 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186792967}) 2024-09-15T20:37:17.585 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3755134456}) 2024-09-15T20:37:17.586 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.586 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848982629}) 2024-09-15T20:37:17.586 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2521744574}) 2024-09-15T20:37:17.587 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.587 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2848982629}) 2024-09-15T20:37:17.587 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2954976014}) 2024-09-15T20:37:17.588 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.588 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480698434}) 2024-09-15T20:37:17.588 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602164030}) 2024-09-15T20:37:17.588 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.588 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1480698434}) 2024-09-15T20:37:17.588 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.584+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=750296976}) 2024-09-15T20:37:17.590 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.590 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223909353}) 2024-09-15T20:37:17.590 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1198076229}) 2024-09-15T20:37:17.590 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.590 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4223909353}) 2024-09-15T20:37:17.590 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=448506745}) 2024-09-15T20:37:17.591 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.591 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=655494621}) 2024-09-15T20:37:17.592 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1669299214}) 2024-09-15T20:37:17.592 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.592 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=655494621}) 2024-09-15T20:37:17.592 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.588+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3552721009}) 2024-09-15T20:37:17.593 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.593 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354458729}) 2024-09-15T20:37:17.593 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2867028523}) 2024-09-15T20:37:17.593 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.594 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2354458729}) 2024-09-15T20:37:17.594 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1643414316}) 2024-09-15T20:37:17.595 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.595 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545310222}) 2024-09-15T20:37:17.595 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1833547248}) 2024-09-15T20:37:17.595 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.595 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2545310222}) 2024-09-15T20:37:17.596 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.592+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3882061068}) 2024-09-15T20:37:17.597 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.597 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1781246453}) 2024-09-15T20:37:17.597 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2734885686}) 2024-09-15T20:37:17.597 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.597 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1781246453}) 2024-09-15T20:37:17.597 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2735847598}) 2024-09-15T20:37:17.598 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.598 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718999824}) 2024-09-15T20:37:17.598 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2343511319}) 2024-09-15T20:37:17.598 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.598 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3718999824}) 2024-09-15T20:37:17.598 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3445295419}) 2024-09-15T20:37:17.600 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.600 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3032262898}) 2024-09-15T20:37:17.600 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=856382979}) 2024-09-15T20:37:17.600 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.600 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3032262898}) 2024-09-15T20:37:17.600 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.596+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2963733639}) 2024-09-15T20:37:17.602 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.602 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837561422}) 2024-09-15T20:37:17.602 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919846451}) 2024-09-15T20:37:17.602 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.602 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=837561422}) 2024-09-15T20:37:17.602 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3157701605}) 2024-09-15T20:37:17.604 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.604 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117992696}) 2024-09-15T20:37:17.604 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3547995009}) 2024-09-15T20:37:17.604 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.604 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3117992696}) 2024-09-15T20:37:17.604 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.600+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1671780913}) 2024-09-15T20:37:17.605 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.605 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265058841}) 2024-09-15T20:37:17.605 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=723876076}) 2024-09-15T20:37:17.605 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.606 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265058841}) 2024-09-15T20:37:17.606 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3197094269}) 2024-09-15T20:37:17.609 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.609 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500069413}) 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2021373687}) 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=500069413}) 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.604+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3780693678}) 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452836691}) 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2351552667}) 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1452836691}) 2024-09-15T20:37:17.610 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377512676}) 2024-09-15T20:37:17.611 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.611 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1370199039,mdsmap=503983409,mgr=3189685207}) 2024-09-15T20:37:17.611 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=93,mdsmap=3,mgr=4} crc {logm=2772110606,mdsmap=503983409,mgr=1257327691}) 2024-09-15T20:37:17.611 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.612 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=96,mdsmap=3,mgr=1} crc {logm=1370199039,mdsmap=503983409,mgr=3189685207}) 2024-09-15T20:37:17.612 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.608+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=140509254,mdsmap=503983409,mgr=1516913497}) 2024-09-15T20:37:17.614 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.614 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=27,osdmap=22} crc {mgr=1631156557,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=1178910840,mon_config_key=4293202365,monmap=3785329752,nvmeofgw=598337686,osd_metadata=2781288008,osd_pg_creating=227486527,osd_snap=191481520,osdmap=2687672549}) 2024-09-15T20:37:17.614 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=27,osdmap=21} crc {mgr=2467901132,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=1716478547,mon_config_key=3551184854,monmap=328434086,nvmeofgw=598337686,osd_metadata=2781288008,osd_pg_creating=227486527,osd_snap=786758009,osdmap=4279455230}) 2024-09-15T20:37:17.614 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.614 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=27,osdmap=22} crc {mgr=1631156557,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=1178910840,mon_config_key=4293202365,monmap=3785329752,nvmeofgw=598337686,osd_metadata=2781288008,osd_pg_creating=227486527,osd_snap=191481520,osdmap=2687672549}) 2024-09-15T20:37:17.614 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=6,mgr_command_descs=1,mgr_metadata=1,mgrstat=14,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=21} crc {mgr=2063128282,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=646550790,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=227486527,osd_snap=786758009,osdmap=254788620}) 2024-09-15T20:37:17.616 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.616 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=722886794}) 2024-09-15T20:37:17.616 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1470194307}) 2024-09-15T20:37:17.616 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.616 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=722886794}) 2024-09-15T20:37:17.616 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.612+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2554693646}) 2024-09-15T20:37:17.619 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.616+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.619 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.616+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2611191425}) 2024-09-15T20:37:17.619 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.616+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2017588277}) 2024-09-15T20:37:17.619 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.616+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.619 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.616+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2611191425}) 2024-09-15T20:37:17.620 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.616+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2745235649}) 2024-09-15T20:37:17.622 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.620+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.622 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.620+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=709312362}) 2024-09-15T20:37:17.622 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.620+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3491549130}) 2024-09-15T20:37:17.622 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.620+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.623 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.620+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=709312362}) 2024-09-15T20:37:17.623 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.620+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1108777359}) 2024-09-15T20:37:17.626 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.624+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.626 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.624+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=2894411515}) 2024-09-15T20:37:17.626 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.624+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=70} crc {osdmap=4127240009}) 2024-09-15T20:37:17.626 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.624+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:37:17.626 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.624+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=70} crc {osdmap=2894411515}) 2024-09-15T20:37:17.627 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:37:17.624+0000 7f650ae13640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=76} crc {osdmap=2317306542}) 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:7462ddf6:o.RoundTripAppendPP (3019 ms) 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RacingRemovePP 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RacingRemovePP (2936 ms) 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP (3005 ms) 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.RoundTripCmpExtPP2 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.RoundTripCmpExtPP2 (3025 ms) 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.PoolEIOFlag 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: setting pool EIO 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: max_success 99, min_failed 100 2024-09-15T20:37:18.557 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.PoolEIOFlag (3964 ms) 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAio.MultiReads 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAio.MultiReads (3023 ms) 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] 32 tests from LibRadosAio (128988 ms total) 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.ReadIntoBufferlist 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioPP.ReadIntoBufferlist (3024 ms) 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.XattrsRoundTripPP 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioPP.XattrsRoundTripPP (9024 ms) 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RmXattrPP 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RmXattrPP (14959 ms) 2024-09-15T20:37:18.558 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioPP.RemoveTestPP 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioPP.RemoveTestPP (3010 ms) 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] 4 tests from LibRadosAioPP (30018 ms total) 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosIoPP.XattrListPP 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosIoPP.XattrListPP (3013 ms) 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] 1 test from LibRadosIoPP (3013 ms total) 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleWritePP 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleWritePP (28000 ms) 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.WaitForSafePP 2024-09-15T20:37:18.559 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.WaitForSafePP (6993 ms) 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP (6987 ms) 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP2 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP2 (7015 ms) 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripPP3 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripPP3 (2988 ms) 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripSparseReadPP 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripSparseReadPP (7047 ms) 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripAppendPP 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripAppendPP (6978 ms) 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsCompletePP 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsCompletePP (6986 ms) 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.IsSafePP 2024-09-15T20:37:18.560 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.IsSafePP (7024 ms) 2024-09-15T20:37:18.561 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ReturnValuePP 2024-09-15T20:37:18.561 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ReturnValuePP (7007 ms) 2024-09-15T20:37:18.561 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushPP 2024-09-15T20:37:18.561 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushPP (6978 ms) 2024-09-15T20:37:18.561 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.FlushAsyncPP 2024-09-15T20:37:18.561 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.FlushAsyncPP (7034 ms) 2024-09-15T20:37:18.561 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP 2024-09-15T20:37:18.561 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP (6916 ms) 2024-09-15T20:37:18.578 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:37:18.579 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient: get_monmap_and_config 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient: build_initial_monmap 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient: monmap: 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:18.703101+0000 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:18.703101+0000 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:18.703 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:18.704 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 1 Processor -- start 2024-09-15T20:37:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 1 -- start start 2024-09-15T20:37:18.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient: init 2024-09-15T20:37:18.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient(hunting): picked mon.noname-b con 0x7fd964135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient(hunting): picked mon.noname-c con 0x7fd9640a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient(hunting): picked mon.noname-a con 0x7fd96413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9640745e0 con 0x7fd96413c220 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96406ecf0 con 0x7fd9640a24e0 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96406efd0 con 0x7fd964135b60 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd96b522640 10 monclient(hunting): authenticate will time out at 1499.587641s 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd9692be640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd964135b60 0x7fd964135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57282/0 (socket says 172.21.15.78:57282) 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd969abf640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd96413c220 0x7fd96413e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50580/0 (socket says 172.21.15.78:50580) 2024-09-15T20:37:18.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.700+0000 7fd9692be640 1 -- 172.21.15.78:0/3590376374 learned_addr learned my addr 172.21.15.78:0/3590376374 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317449019 0 0) 0x7fd96406efd0 con 0x7fd964135b60 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): my global_id is 84350 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd950003530 con 0x7fd964135b60 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2786655126 0 0) 0x7fd9640745e0 con 0x7fd96413c220 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): my global_id is 94131 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96406efd0 con 0x7fd96413c220 2024-09-15T20:37:18.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 637573465 0 0) 0x7fd96406ecf0 con 0x7fd9640a24e0 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient(hunting): my global_id is 23410 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9640745e0 con 0x7fd9640a24e0 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2427098876 0 0) 0x7fd96406efd0 con 0x7fd96413c220 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 >> v1:172.21.15.78:6790/0 conn(0x7fd964135b60 legacy=0x7fd964135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 >> v1:172.21.15.146:6789/0 conn(0x7fd9640a24e0 legacy=0x7fd96413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 monclient: found mon.noname-a 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:18.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd964136650 con 0x7fd96413c220 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: _finish_auth 0 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 5 monclient: authenticate success, global_id 94131 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd95c001a00 con 0x7fd96413c220 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: dump: 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:18.714 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd95c001ba0 con 0x7fd96413c220 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 1 -- 172.21.15.78:0/3590376374 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd95c001f70 con 0x7fd96413c220 2024-09-15T20:37:18.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 4 set_mon_vals no callback set 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd95bfff640 10 monclient: dump: 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:18.716 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: shutdown 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- 172.21.15.78:0/3590376374 >> v1:172.21.15.78:6789/0 conn(0x7fd96413c220 legacy=0x7fd96413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- 172.21.15.78:0/3590376374 shutdown_connections 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- 172.21.15.78:0/3590376374 >> 172.21.15.78:0/3590376374 conn(0x7fd9640a63c0 msgr2=0x7fd9640a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- 172.21.15.78:0/3590376374 shutdown_connections 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- 172.21.15.78:0/3590376374 wait complete. 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: build_initial_monmap 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: monmap: 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:18.717 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 librados: starting msgr at 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 librados: starting objecter 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 Processor -- start 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- start start 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 librados: setting wanted keys 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 librados: calling monclient init 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: init 2024-09-15T20:37:18.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient(hunting): picked mon.noname-a con 0x7fd9640a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient(hunting): picked mon.noname-b con 0x7fd964135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient(hunting): picked mon.noname-c con 0x7fd96413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9641395c0 con 0x7fd9640a24e0 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9641398a0 con 0x7fd964135b60 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd96413f9e0 con 0x7fd96413c220 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient(hunting): authenticate will time out at 1499.590951s 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd9692be640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd9640a24e0 0x7fd96409d6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50592/0 (socket says 172.21.15.78:50592) 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd969abf640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd96413c220 0x7fd9641d4c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57286/0 (socket says 172.21.15.78:57286) 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd9692be640 1 -- 172.21.15.78:0/1167426418 learned_addr learned my addr 172.21.15.78:0/1167426418 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 client.?.objecter ms_handle_connect 0x7fd96413c220 2024-09-15T20:37:18.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 client.?.objecter ms_handle_connect 0x7fd9640a24e0 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3626790133 0 0) 0x7fd96413f9e0 con 0x7fd96413c220 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): my global_id is 84353 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd948004ec0 con 0x7fd96413c220 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 client.?.objecter ms_handle_connect 0x7fd964135b60 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 554406410 0 0) 0x7fd9641395c0 con 0x7fd9640a24e0 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:18.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): my global_id is 94134 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96413f9e0 con 0x7fd9640a24e0 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4175704535 0 0) 0x7fd948004ec0 con 0x7fd96413c220 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9641395c0 con 0x7fd96413c220 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd95c0018c0 con 0x7fd96413c220 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): dump: 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.721 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4048310863 0 0) 0x7fd96413f9e0 con 0x7fd9640a24e0 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd948004ec0 con 0x7fd9640a24e0 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4061822288 0 0) 0x7fd9641398a0 con 0x7fd964135b60 2024-09-15T20:37:18.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): my global_id is 23413 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd96413f9e0 con 0x7fd964135b60 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd9540019f0 con 0x7fd9640a24e0 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient(hunting): dump: 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.723 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1670602430 0 0) 0x7fd948004ec0 con 0x7fd9640a24e0 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 >> v1:172.21.15.78:6790/0 conn(0x7fd96413c220 legacy=0x7fd9641d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 >> v1:172.21.15.146:6789/0 conn(0x7fd964135b60 legacy=0x7fd96409de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 monclient: found mon.a 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:37:18.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd96409eef0 con 0x7fd9640a24e0 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient: _finish_auth 0 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd959ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 5 monclient: authenticate success, global_id 94134 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: _renew_subs 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd964139b80 con 0x7fd9640a24e0 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 client.94134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: _renew_subs 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.704+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd96409e510 con 0x7fd9640a24e0 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd96b522640 1 librados: init done 2024-09-15T20:37:18.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd9540047f0 con 0x7fd9640a24e0 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd954004b40 con 0x7fd9640a24e0 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd95b7fe640 4 set_mon_vals no callback set 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 10 monclient: dump: 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:18.726 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd96b522640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd96b522640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd96b522640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd944005090 con 0x7fd9640a24e0 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd954010bb0 con 0x7fd9640a24e0 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(263..263 src has 1..263) ==== 5364+0+0 (unknown 1454912694 0 0) 0x7fd95404b710 con 0x7fd9640a24e0 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 10 client.94134.objecter ms_dispatch 0x7fd964000ba0 osd_map(263..263 src has 1..263) 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 3 client.94134.objecter handle_osd_map got epochs [263,263] > 0 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 3 client.94134.objecter handle_osd_map decoding full epoch 263 2024-09-15T20:37:18.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 20 client.94134.objecter dump_active .. 0 homeless 2024-09-15T20:37:18.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd968abd640 10 monclient: get_auth_request con 0x7fd94803e1c0 auth_method 0 2024-09-15T20:37:18.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.708+0000 7fd959ffb640 10 client.94134.objecter ms_handle_connect 0x7fd94803e1c0 2024-09-15T20:37:18.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.712+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd9540169a0 con 0x7fd9640a24e0 2024-09-15T20:37:18.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.712+0000 7fd959ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:18.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.712+0000 7fd959ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:18.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.932+0000 7fd96b522640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:37:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.932+0000 7fd96b522640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:37:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.932+0000 7fd96b522640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:37:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.932+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd944005380 con 0x7fd9640a24e0 2024-09-15T20:37:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd959ffb640 1 -- 172.21.15.78:0/1167426418 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 147467806) 0x7fd9540162a0 con 0x7fd9640a24e0 2024-09-15T20:37:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd959ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:37:18.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd959ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:18.937 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:37:18.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 10 librados: watch_flush enter 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 10 librados: watch_flush exit 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 20 client.94134.objecter shutdown clearing up homeless session... 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 10 client.94134.objecter successfully canceled tick 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 >> v1:172.21.15.146:6812/435598642 conn(0x7fd94803e1c0 legacy=0x7fd948040680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 10 monclient: shutdown 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 >> v1:172.21.15.78:6789/0 conn(0x7fd9640a24e0 legacy=0x7fd96409d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 shutdown_connections 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 >> 172.21.15.78:0/1167426418 conn(0x7fd9640a63c0 msgr2=0x7fd96413eec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 shutdown_connections 2024-09-15T20:37:18.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 1 -- 172.21.15.78:0/1167426418 wait complete. 2024-09-15T20:37:18.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:18.936+0000 7fd96b522640 1 librados: shutdown 2024-09-15T20:37:18.953 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:37:18.953 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:37:18.953 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:37:18.953 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:37:19.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: get_monmap_and_config 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: build_initial_monmap 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: monmap: 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:19.061025+0000 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:19.061025+0000 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:19.061 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:19.062 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:19.062 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 Processor -- start 2024-09-15T20:37:19.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- start start 2024-09-15T20:37:19.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: init 2024-09-15T20:37:19.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:19.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:37:19.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient(hunting): picked mon.noname-c con 0x7f87f4137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:19.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient(hunting): picked mon.noname-a con 0x7f87f40a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:19.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient(hunting): picked mon.noname-b con 0x7f87f413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f40a6420 con 0x7f87f40a9610 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f40a6700 con 0x7f87f4137ce0 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f40a2110 con 0x7f87f413e5f0 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient(hunting): authenticate will time out at 1499.945619s 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f94be640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f87f40a9610 0x7f87f413ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50604/0 (socket says 172.21.15.78:50604) 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f94be640 1 -- 172.21.15.78:0/320023022 learned_addr learned my addr 172.21.15.78:0/320023022 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3692034307 0 0) 0x7f87f40a2110 con 0x7f87f413e5f0 2024-09-15T20:37:19.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient(hunting): my global_id is 84356 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87dc003530 con 0x7f87f413e5f0 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1307128922 0 0) 0x7f87f40a6420 con 0x7f87f40a9610 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient(hunting): my global_id is 94137 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f40a2110 con 0x7f87f40a9610 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1115089593 0 0) 0x7f87dc003530 con 0x7f87f413e5f0 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 >> v1:172.21.15.146:6789/0 conn(0x7f87f4137ce0 legacy=0x7f87f41380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 >> v1:172.21.15.78:6789/0 conn(0x7f87f40a9610 legacy=0x7f87f413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: _finish_hunting 0 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 monclient: found mon.noname-b 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f40a4bc0 con 0x7f87f413e5f0 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: _finish_auth 0 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: _check_auth_tickets 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87e40016a0 con 0x7f87f413e5f0 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 5 monclient: authenticate success, global_id 84356 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: dump: 2024-09-15T20:37:19.079 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87e40018c0 con 0x7f87f413e5f0 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:19.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 4 set_mon_vals no callback set 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 1 -- 172.21.15.78:0/320023022 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87e4001c90 con 0x7f87f413e5f0 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87f8cbd640 10 monclient: dump: 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: shutdown 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- 172.21.15.78:0/320023022 >> v1:172.21.15.78:6790/0 conn(0x7f87f413e5f0 legacy=0x7f87f41409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- 172.21.15.78:0/320023022 shutdown_connections 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- 172.21.15.78:0/320023022 >> 172.21.15.78:0/320023022 conn(0x7f87f40aa6c0 msgr2=0x7f87f40aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- 172.21.15.78:0/320023022 shutdown_connections 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- 172.21.15.78:0/320023022 wait complete. 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: build_initial_monmap 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: monmap: 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:19.082 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 librados: starting msgr at 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 librados: starting objecter 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 Processor -- start 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 -- start start 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 librados: setting wanted keys 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 1 librados: calling monclient init 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.060+0000 7f87fbf23640 10 monclient: init 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:37:19.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient(hunting): picked mon.noname-a con 0x7f87f40a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient(hunting): picked mon.noname-b con 0x7f87f4137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient(hunting): picked mon.noname-c con 0x7f87f413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f40a0510 con 0x7f87f40a9610 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f41cdd20 con 0x7f87f4137ce0 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f87f40a14b0 con 0x7f87f413e5f0 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient(hunting): authenticate will time out at 1499.948348s 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87f9cbf640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f87f40a9610 0x7f87f41ccf00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50612/0 (socket says 172.21.15.78:50612) 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87f9cbf640 1 -- 172.21.15.78:0/197751000 learned_addr learned my addr 172.21.15.78:0/197751000 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.?.objecter ms_handle_connect 0x7f87f413e5f0 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.?.objecter ms_handle_connect 0x7f87f40a9610 2024-09-15T20:37:19.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4107164495 0 0) 0x7f87f40a14b0 con 0x7f87f413e5f0 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): my global_id is 84359 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87e0004ec0 con 0x7f87f413e5f0 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4080470698 0 0) 0x7f87f40a0510 con 0x7f87f40a9610 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): my global_id is 94140 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f40a14b0 con 0x7f87f40a9610 2024-09-15T20:37:19.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.?.objecter ms_handle_connect 0x7f87f4137ce0 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2274394321 0 0) 0x7f87e0004ec0 con 0x7f87f413e5f0 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87f40a0510 con 0x7f87f413e5f0 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87e4001c10 con 0x7f87f413e5f0 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): dump: 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.086 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1428152404 0 0) 0x7f87f40a14b0 con 0x7f87f40a9610 2024-09-15T20:37:19.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87e0004ec0 con 0x7f87f40a9610 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87ec001a00 con 0x7f87f40a9610 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): dump: 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.088 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3005484011 0 0) 0x7f87f41cdd20 con 0x7f87f4137ce0 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): my global_id is 23419 2024-09-15T20:37:19.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f40a14b0 con 0x7f87f4137ce0 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 476510406 0 0) 0x7f87f40a0510 con 0x7f87f413e5f0 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 >> v1:172.21.15.146:6789/0 conn(0x7f87f4137ce0 legacy=0x7f87f41cd610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 >> v1:172.21.15.78:6789/0 conn(0x7f87f40a9610 legacy=0x7f87f41ccf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 monclient: found mon.c 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87f413dbb0 con 0x7f87f413e5f0 2024-09-15T20:37:19.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: _finish_auth 0 2024-09-15T20:37:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:37:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 5 monclient: authenticate success, global_id 84359 2024-09-15T20:37:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient: _renew_subs 2024-09-15T20:37:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f87f409ff50 con 0x7f87f413e5f0 2024-09-15T20:37:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 client.84359.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:19.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient: _renew_subs 2024-09-15T20:37:19.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:19.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f87f409dcd0 con 0x7f87f413e5f0 2024-09-15T20:37:19.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 1 librados: init done 2024-09-15T20:37:19.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f87e4001190 con 0x7f87f413e5f0 2024-09-15T20:37:19.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:19.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87e4003970 con 0x7f87f413e5f0 2024-09-15T20:37:19.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ebfff640 4 set_mon_vals no callback set 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: dump: 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.093 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.094 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.094 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.094 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.094 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:19.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f87d40065d0 con 0x7f87d4002870 2024-09-15T20:37:19.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.84359.objecter ms_handle_connect 0x7f87d4002870 2024-09-15T20:37:19.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.84359.objecter resend_mon_ops 2024-09-15T20:37:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f87e400f9e0 con 0x7f87f413e5f0 2024-09-15T20:37:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(263..263 src has 1..263) ==== 5364+0+0 (unknown 1454912694 0 0) 0x7f87e404a490 con 0x7f87f413e5f0 2024-09-15T20:37:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.84359.objecter ms_dispatch 0x7f87f4000ba0 osd_map(263..263 src has 1..263) 2024-09-15T20:37:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 3 client.84359.objecter handle_osd_map got epochs [263,263] > 0 2024-09-15T20:37:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 3 client.84359.objecter handle_osd_map decoding full epoch 263 2024-09-15T20:37:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 20 client.84359.objecter dump_active .. 0 homeless 2024-09-15T20:37:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1273678190 0 0) 0x7f87d40065d0 con 0x7f87d4002870 2024-09-15T20:37:19.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient(hunting): my global_id is 94143 2024-09-15T20:37:19.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87f40a0510 con 0x7f87d4002870 2024-09-15T20:37:19.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87f94be640 10 monclient: get_auth_request con 0x7f87e003e1f0 auth_method 0 2024-09-15T20:37:19.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2348175392 0 0) 0x7f87f40a0510 con 0x7f87d4002870 2024-09-15T20:37:19.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87d40065d0 con 0x7f87d4002870 2024-09-15T20:37:19.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87ec003a70 con 0x7f87d4002870 2024-09-15T20:37:19.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:19.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: dump: 2024-09-15T20:37:19.097 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.097 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.097 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.098 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.098 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.098 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.098 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.098 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.098 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.098 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2351615780 0 0) 0x7f87d40065d0 con 0x7f87d4002870 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f87d400a170 con 0x7f87d4002870 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 client.84359.objecter ms_handle_connect 0x7f87e003e1f0 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f87ec0057d0 con 0x7f87d4002870 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 >> v1:172.21.15.78:6789/0 conn(0x7f87d4002870 legacy=0x7f87d4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.064+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 >> v1:172.21.15.78:6789/0 conn(0x7f87d4002870 legacy=0x7f87d4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:19.099 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:37:19.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f87d4009e60 con 0x7f87d4009120 2024-09-15T20:37:19.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 client.84359.objecter ms_handle_connect 0x7f87d4009120 2024-09-15T20:37:19.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 client.84359.objecter resend_mon_ops 2024-09-15T20:37:19.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 447275359 0 0) 0x7f87d4009e60 con 0x7f87d4009120 2024-09-15T20:37:19.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient(hunting): my global_id is 94146 2024-09-15T20:37:19.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f87d40065d0 con 0x7f87d4009120 2024-09-15T20:37:19.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2472894154 0 0) 0x7f87d40065d0 con 0x7f87d4009120 2024-09-15T20:37:19.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f87d4009e60 con 0x7f87d4009120 2024-09-15T20:37:19.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f87ec004160 con 0x7f87d4009120 2024-09-15T20:37:19.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:19.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient: dump: 2024-09-15T20:37:19.102 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.102 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.102 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.102 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.102 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.102 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.102 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.102 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.103 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.103 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2916324639 0 0) 0x7f87d4009e60 con 0x7f87d4009120 2024-09-15T20:37:19.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f87d400c400 con 0x7f87d4009120 2024-09-15T20:37:19.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3971 (unknown 0 0 253121324) 0x7f87ec003f80 con 0x7f87d4009120 2024-09-15T20:37:19.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:37:19.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:19.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 >> v1:172.21.15.78:6789/0 conn(0x7f87d4009120 legacy=0x7f87d40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87ea7fc640 1 -- 172.21.15.78:0/197751000 >> v1:172.21.15.78:6789/0 conn(0x7f87d4009120 legacy=0x7f87d40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:19.104 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:37:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:19.104 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:37:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 24, 2024-09-15T20:37:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 2735, 2024-09-15T20:37:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:37:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:37:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:37:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:37:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.105 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.106 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.107 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.108 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:37:19.109 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.110 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.111 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:19.112 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.113 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:37:19.114 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.115 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.116 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.117 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:37:19.117 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:37:19.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 10 librados: watch_flush enter 2024-09-15T20:37:19.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 10 librados: watch_flush exit 2024-09-15T20:37:19.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 20 client.84359.objecter shutdown clearing up homeless session... 2024-09-15T20:37:19.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 10 client.84359.objecter successfully canceled tick 2024-09-15T20:37:19.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 >> v1:172.21.15.146:6812/435598642 conn(0x7f87e003e1f0 legacy=0x7f87e00406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 10 monclient: shutdown 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 >> v1:172.21.15.78:6790/0 conn(0x7f87f413e5f0 legacy=0x7f87f41d5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fa4c0640 1 -- 172.21.15.78:0/197751000 reap_dead start 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fa4c0640 1 -- 172.21.15.78:0/197751000 reap_dead start 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 shutdown_connections 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 >> 172.21.15.78:0/197751000 conn(0x7f87f40aa6c0 msgr2=0x7f87f4130f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 shutdown_connections 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 1 -- 172.21.15.78:0/197751000 wait complete. 2024-09-15T20:37:19.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.072+0000 7f87fbf23640 1 librados: shutdown 2024-09-15T20:37:19.121 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:37:19.121 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:37:19.121 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:37:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient: get_monmap_and_config 2024-09-15T20:37:19.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient: build_initial_monmap 2024-09-15T20:37:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient: monmap: 2024-09-15T20:37:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:19.234 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:19.233841+0000 2024-09-15T20:37:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:19.233841+0000 2024-09-15T20:37:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:19.235 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:19.235 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 1 Processor -- start 2024-09-15T20:37:19.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 1 -- start start 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient: init 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient(hunting): picked mon.noname-c con 0x7fb7f40a5680 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient(hunting): picked mon.noname-b con 0x7fb7f40a6170 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f413b690 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f40744b0 con 0x7fb7f413b690 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f40a0ca0 con 0x7fb7f40a5680 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f409f4f0 con 0x7fb7f40a6170 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:19.246 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f9941640 10 monclient(hunting): authenticate will time out at 1500.119282s 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f27fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb7f40a6170 0x7fb7f4139f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57312/0 (socket says 172.21.15.78:57312) 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f27fc640 1 -- 172.21.15.78:0/614479886 learned_addr learned my addr 172.21.15.78:0/614479886 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461518495 0 0) 0x7fb7f409f4f0 con 0x7fb7f40a6170 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 10 monclient(hunting): my global_id is 84362 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7dc003530 con 0x7fb7f40a6170 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3954392620 0 0) 0x7fb7ec001050 con 0x7fb7f413b690 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 10 monclient(hunting): my global_id is 94149 2024-09-15T20:37:19.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7f409f4f0 con 0x7fb7f413b690 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 454100786 0 0) 0x7fb7dc003530 con 0x7fb7f40a6170 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.231+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 >> v1:172.21.15.146:6789/0 conn(0x7fb7f40a5680 legacy=0x7fb7f40a5a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 >> v1:172.21.15.78:6789/0 conn(0x7fb7f413b690 legacy=0x7fb7f413da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 1 monclient: found mon.noname-b 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f409f7d0 con 0x7fb7f40a6170 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: _finish_auth 0 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb7e80016a0 con 0x7fb7f40a6170 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 5 monclient: authenticate success, global_id 84362 2024-09-15T20:37:19.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: dump: 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.249 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7e80018c0 con 0x7fb7f40a6170 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 1 -- 172.21.15.78:0/614479886 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb7e8001c90 con 0x7fb7f40a6170 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f1ffb640 10 monclient: dump: 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.250 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 4 set_mon_vals no callback set 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: shutdown 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- 172.21.15.78:0/614479886 >> v1:172.21.15.78:6790/0 conn(0x7fb7f40a6170 legacy=0x7fb7f4139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- 172.21.15.78:0/614479886 shutdown_connections 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- 172.21.15.78:0/614479886 >> 172.21.15.78:0/614479886 conn(0x7fb7f40a6ee0 msgr2=0x7fb7f40a3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- 172.21.15.78:0/614479886 shutdown_connections 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- 172.21.15.78:0/614479886 wait complete. 2024-09-15T20:37:19.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: build_initial_monmap 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: monmap: 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 librados: starting msgr at 2024-09-15T20:37:19.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 librados: starting objecter 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 Processor -- start 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- start start 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 librados: setting wanted keys 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 librados: calling monclient init 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: init 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient(hunting): picked mon.noname-b con 0x7fb7f40a5680 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient(hunting): picked mon.noname-c con 0x7fb7f40a6170 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient(hunting): picked mon.noname-a con 0x7fb7f413b690 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f4135bc0 con 0x7fb7f413b690 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f4074c30 con 0x7fb7f40a5680 2024-09-15T20:37:19.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7f413ad50 con 0x7fb7f40a6170 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient(hunting): authenticate will time out at 1500.122297s 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f27fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb7f40a6170 0x7fb7f41d1e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57316/0 (socket says 172.21.15.78:57316) 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb7f413b690 0x7fb7f41d5550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50654/0 (socket says 172.21.15.78:50654) 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f27fc640 1 -- 172.21.15.78:0/999375931 learned_addr learned my addr 172.21.15.78:0/999375931 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 client.?.objecter ms_handle_connect 0x7fb7f413b690 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 client.?.objecter ms_handle_connect 0x7fb7f40a6170 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3044864693 0 0) 0x7fb7f4135bc0 con 0x7fb7f413b690 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): my global_id is 94152 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7e0004ef0 con 0x7fb7f413b690 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3000000133 0 0) 0x7fb7f413ad50 con 0x7fb7f40a6170 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): my global_id is 84365 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7f4135bc0 con 0x7fb7f40a6170 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 client.?.objecter ms_handle_connect 0x7fb7f40a5680 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 955523253 0 0) 0x7fb7e0004ef0 con 0x7fb7f413b690 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7f413ad50 con 0x7fb7f413b690 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb7ec001a40 con 0x7fb7f413b690 2024-09-15T20:37:19.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): dump: 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:19.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3345820605 0 0) 0x7fb7f4135bc0 con 0x7fb7f40a6170 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7e0004ef0 con 0x7fb7f40a6170 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb7e8001c10 con 0x7fb7f40a6170 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): dump: 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.257 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3696262145 0 0) 0x7fb7f4074c30 con 0x7fb7f40a5680 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient(hunting): my global_id is 23425 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7f4135bc0 con 0x7fb7f40a5680 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3368531918 0 0) 0x7fb7f413ad50 con 0x7fb7f413b690 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 >> v1:172.21.15.78:6790/0 conn(0x7fb7f40a6170 legacy=0x7fb7f41d1e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 >> v1:172.21.15.146:6789/0 conn(0x7fb7f40a5680 legacy=0x7fb7f4135230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient: _finish_hunting 0 2024-09-15T20:37:19.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 monclient: found mon.a 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7f4123920 con 0x7fb7f413b690 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient: _finish_auth 0 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient: _check_auth_tickets 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 5 monclient: authenticate success, global_id 94152 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: _renew_subs 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb7f4123c00 con 0x7fb7f413b690 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 client.94152.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: _renew_subs 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:37:19.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb7f4122400 con 0x7fb7f413b690 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 librados: init done 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb7ec001c60 con 0x7fb7f413b690 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb7ec003a70 con 0x7fb7f413b690 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f17fa640 4 set_mon_vals no callback set 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7dbfff640 10 monclient: dump: 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.260 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.235+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb7d40065d0 con 0x7fb7d4002870 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 client.94152.objecter ms_handle_connect 0x7fb7d4002870 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 client.94152.objecter resend_mon_ops 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb7ec003c90 con 0x7fb7f413b690 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(263..263 src has 1..263) ==== 5364+0+0 (unknown 1454912694 0 0) 0x7fb7ec04b6c0 con 0x7fb7f413b690 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 client.94152.objecter ms_dispatch 0x7fb7f4000ba0 osd_map(263..263 src has 1..263) 2024-09-15T20:37:19.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 3 client.94152.objecter handle_osd_map got epochs [263,263] > 0 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 3 client.94152.objecter handle_osd_map decoding full epoch 263 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 20 client.94152.objecter dump_active .. 0 homeless 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1745013189 0 0) 0x7fb7d40065d0 con 0x7fb7d4002870 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient(hunting): my global_id is 84368 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7f413ad50 con 0x7fb7d4002870 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2125296994 0 0) 0x7fb7f413ad50 con 0x7fb7d4002870 2024-09-15T20:37:19.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7d40065d0 con 0x7fb7d4002870 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb7e4001aa0 con 0x7fb7d4002870 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7f27fc640 10 monclient: get_auth_request con 0x7fb7e003e240 auth_method 0 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient: dump: 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.263 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 344117530 0 0) 0x7fb7d40065d0 con 0x7fb7d4002870 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb7d400a170 con 0x7fb7d4002870 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 client.94152.objecter ms_handle_connect 0x7fb7e003e240 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb7e4001d40 con 0x7fb7d4002870 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 >> v1:172.21.15.78:6790/0 conn(0x7fb7d4002870 legacy=0x7fb7d4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.239+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 >> v1:172.21.15.78:6790/0 conn(0x7fb7d4002870 legacy=0x7fb7d4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7f9941640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb7d4009e80 con 0x7fb7d4009140 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 client.94152.objecter ms_handle_connect 0x7fb7d4009140 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 client.94152.objecter resend_mon_ops 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1818551998 0 0) 0x7fb7d4009e80 con 0x7fb7d4009140 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 monclient(hunting): my global_id is 84371 2024-09-15T20:37:19.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb7d40065d0 con 0x7fb7d4009140 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4121697956 0 0) 0x7fb7d40065d0 con 0x7fb7d4009140 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb7d4009e80 con 0x7fb7d4009140 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb7e4003b20 con 0x7fb7d4009140 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.243+0000 7fb7dbfff640 10 monclient: dump: 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.267 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4039503006 0 0) 0x7fb7d4009e80 con 0x7fb7d4009140 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb7d400c420 con 0x7fb7d4009140 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 3906638275) 0x7fb7e4003ab0 con 0x7fb7d4009140 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7dbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7dbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 >> v1:172.21.15.78:6790/0 conn(0x7fb7d4009140 legacy=0x7fb7d40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7dbfff640 1 -- 172.21.15.78:0/999375931 >> v1:172.21.15.78:6790/0 conn(0x7fb7d4009140 legacy=0x7fb7d40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:19.268 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 24, 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 110934, 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:19.269 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.270 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.271 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:37:19.272 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.273 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.274 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.275 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.276 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:37:19.277 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:37:19.278 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:37:19.279 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:37:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.280 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:37:19.280 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 10 librados: watch_flush enter 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 10 librados: watch_flush exit 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 20 client.94152.objecter shutdown clearing up homeless session... 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 10 client.94152.objecter successfully canceled tick 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 >> v1:172.21.15.146:6812/435598642 conn(0x7fb7e003e240 legacy=0x7fb7e0040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 10 monclient: shutdown 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 >> v1:172.21.15.78:6789/0 conn(0x7fb7f413b690 legacy=0x7fb7f41d5550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f37fe640 1 -- 172.21.15.78:0/999375931 reap_dead start 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f37fe640 1 -- 172.21.15.78:0/999375931 reap_dead start 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 shutdown_connections 2024-09-15T20:37:19.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 >> 172.21.15.78:0/999375931 conn(0x7fb7f40a6ee0 msgr2=0x7fb7f40a10e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 shutdown_connections 2024-09-15T20:37:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 1 -- 172.21.15.78:0/999375931 wait complete. 2024-09-15T20:37:19.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.247+0000 7fb7f9941640 1 librados: shutdown 2024-09-15T20:37:19.283 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:37:19.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:37:19.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:37:19.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient: get_monmap_and_config 2024-09-15T20:37:19.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient: build_initial_monmap 2024-09-15T20:37:19.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient: monmap: 2024-09-15T20:37:19.397 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:19.397894+0000 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:19.397894+0000 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:19.398 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 1 Processor -- start 2024-09-15T20:37:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 1 -- start start 2024-09-15T20:37:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient: init 2024-09-15T20:37:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:37:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient(hunting): picked mon.noname-b con 0x7ff84c135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient(hunting): picked mon.noname-a con 0x7ff84c0a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient(hunting): picked mon.noname-c con 0x7ff84c13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84c0745e0 con 0x7ff84c0a24e0 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84c06ecf0 con 0x7ff84c13c220 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84c06efd0 con 0x7ff84c135b60 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient(hunting): authenticate will time out at 1500.282485s 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff84affd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff84c135b60 0x7ff84c135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:57342/0 (socket says 172.21.15.78:57342) 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff84affd640 1 -- 172.21.15.78:0/2078548143 learned_addr learned my addr 172.21.15.78:0/2078548143 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 232115145 0 0) 0x7ff84c0745e0 con 0x7ff84c0a24e0 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient(hunting): my global_id is 94155 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff838003530 con 0x7ff84c0a24e0 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 543440680 0 0) 0x7ff84c06efd0 con 0x7ff84c135b60 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient(hunting): my global_id is 84374 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff84c0745e0 con 0x7ff84c135b60 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4077645825 0 0) 0x7ff838003530 con 0x7ff84c0a24e0 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 >> v1:172.21.15.78:6790/0 conn(0x7ff84c135b60 legacy=0x7ff84c135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 >> v1:172.21.15.146:6789/0 conn(0x7ff84c13c220 legacy=0x7ff84c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 monclient: found mon.noname-a 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:19.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c136650 con 0x7ff84c0a24e0 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: _finish_auth 0 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff83c001a20 con 0x7ff84c0a24e0 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 5 monclient: authenticate success, global_id 94155 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: dump: 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.411 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff83c001bc0 con 0x7ff84c0a24e0 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:19.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 1 -- 172.21.15.78:0/2078548143 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff83c001fd0 con 0x7ff84c0a24e0 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff8514d7640 4 set_mon_vals no callback set 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.395+0000 7ff849ffb640 10 monclient: dump: 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: shutdown 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- 172.21.15.78:0/2078548143 >> v1:172.21.15.78:6789/0 conn(0x7ff84c0a24e0 legacy=0x7ff84c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- 172.21.15.78:0/2078548143 shutdown_connections 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- 172.21.15.78:0/2078548143 >> 172.21.15.78:0/2078548143 conn(0x7ff84c0a63c0 msgr2=0x7ff84c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- 172.21.15.78:0/2078548143 shutdown_connections 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- 172.21.15.78:0/2078548143 wait complete. 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: build_initial_monmap 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: monmap: 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 librados: starting msgr at 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 librados: starting objecter 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 Processor -- start 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- start start 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 librados: setting wanted keys 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 librados: calling monclient init 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: init 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient(hunting): picked mon.noname-b con 0x7ff84c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient(hunting): picked mon.noname-a con 0x7ff84c135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient(hunting): picked mon.noname-c con 0x7ff84c13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84c134fb0 con 0x7ff84c135b60 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84c073590 con 0x7ff84c0a24e0 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff84c138630 con 0x7ff84c13c220 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient(hunting): authenticate will time out at 1500.285870s 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff84a7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff84c135b60 0x7ff84c072440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:50680/0 (socket says 172.21.15.78:50680) 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff84a7fc640 1 -- 172.21.15.78:0/329159965 learned_addr learned my addr 172.21.15.78:0/329159965 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 client.?.objecter ms_handle_connect 0x7ff84c13c220 2024-09-15T20:37:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 client.?.objecter ms_handle_connect 0x7ff84c135b60 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 client.?.objecter ms_handle_connect 0x7ff84c0a24e0 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2828897371 0 0) 0x7ff84c138630 con 0x7ff84c13c220 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): my global_id is 84377 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff82c004ef0 con 0x7ff84c13c220 2024-09-15T20:37:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3080827685 0 0) 0x7ff84c134fb0 con 0x7ff84c135b60 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): my global_id is 94158 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff84c138630 con 0x7ff84c135b60 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2241259321 0 0) 0x7ff84c073590 con 0x7ff84c0a24e0 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): my global_id is 23431 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff84c134fb0 con 0x7ff84c0a24e0 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 298470427 0 0) 0x7ff82c004ef0 con 0x7ff84c13c220 2024-09-15T20:37:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff84c073590 con 0x7ff84c13c220 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff8440018c0 con 0x7ff84c13c220 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): dump: 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3221289217 0 0) 0x7ff84c138630 con 0x7ff84c135b60 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff82c004ef0 con 0x7ff84c135b60 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff83c001f10 con 0x7ff84c135b60 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient(hunting): dump: 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 345775822 0 0) 0x7ff84c073590 con 0x7ff84c13c220 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 >> v1:172.21.15.146:6789/0 conn(0x7ff84c0a24e0 legacy=0x7ff84c071d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 >> v1:172.21.15.78:6789/0 conn(0x7ff84c135b60 legacy=0x7ff84c072440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient: _finish_hunting 0 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 monclient: found mon.c 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff84c137d90 con 0x7ff84c13c220 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient: _finish_auth 0 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient: _check_auth_tickets 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 5 monclient: authenticate success, global_id 84377 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: _renew_subs 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff84c1368d0 con 0x7ff84c13c220 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 client.84377.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: _renew_subs 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff84c072b50 con 0x7ff84c13c220 2024-09-15T20:37:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 librados: init done 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff844003600 con 0x7ff84c13c220 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff844003950 con 0x7ff84c13c220 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff848ff9640 4 set_mon_vals no callback set 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 10 monclient: dump: 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff8300065d0 con 0x7ff830002870 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.399+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff84400f9c0 con 0x7ff84c13c220 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(263..263 src has 1..263) ==== 5364+0+0 (unknown 1454912694 0 0) 0x7ff84404a560 con 0x7ff84c13c220 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 client.84377.objecter ms_dispatch 0x7ff84c000ba0 osd_map(263..263 src has 1..263) 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 3 client.84377.objecter handle_osd_map got epochs [263,263] > 0 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 3 client.84377.objecter handle_osd_map decoding full epoch 263 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 20 client.84377.objecter dump_active .. 0 homeless 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 client.84377.objecter ms_handle_connect 0x7ff830002870 2024-09-15T20:37:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 client.84377.objecter resend_mon_ops 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff84a7fc640 10 monclient: get_auth_request con 0x7ff82c03e1e0 auth_method 0 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030583824 0 0) 0x7ff8300065d0 con 0x7ff830002870 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient(hunting): my global_id is 23434 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff84c073590 con 0x7ff830002870 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 client.84377.objecter ms_handle_connect 0x7ff82c03e1e0 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 817127417 0 0) 0x7ff84c073590 con 0x7ff830002870 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff8300065d0 con 0x7ff830002870 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff834001220 con 0x7ff830002870 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient: dump: 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 935880284 0 0) 0x7ff8300065d0 con 0x7ff830002870 2024-09-15T20:37:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff83000a170 con 0x7ff830002870 2024-09-15T20:37:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff834001d10 con 0x7ff830002870 2024-09-15T20:37:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 >> v1:172.21.15.146:6789/0 conn(0x7ff830002870 legacy=0x7ff830004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.403+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 >> v1:172.21.15.146:6789/0 conn(0x7ff830002870 legacy=0x7ff830004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:19.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:37:19.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff830009e80 con 0x7ff830009140 2024-09-15T20:37:19.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 client.84377.objecter ms_handle_connect 0x7ff830009140 2024-09-15T20:37:19.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 client.84377.objecter resend_mon_ops 2024-09-15T20:37:19.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3904653804 0 0) 0x7ff830009e80 con 0x7ff830009140 2024-09-15T20:37:19.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient(hunting): _negotiate 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient(hunting): my global_id is 23437 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff8300065d0 con 0x7ff830009140 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2415516190 0 0) 0x7ff8300065d0 con 0x7ff830009140 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff830009e80 con 0x7ff830009140 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff834004990 con 0x7ff830009140 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient: dump: 2024-09-15T20:37:19.431 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3092653829 0 0) 0x7ff830009e80 con 0x7ff830009140 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff83000c400 con 0x7ff830009140 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 763067850) 0x7ff834005ea0 con 0x7ff830009140 2024-09-15T20:37:19.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 >> v1:172.21.15.146:6789/0 conn(0x7ff830009140 legacy=0x7ff8300095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8437fe640 1 -- 172.21.15.78:0/329159965 >> v1:172.21.15.146:6789/0 conn(0x7ff830009140 legacy=0x7ff8300095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 24, 2024-09-15T20:37:19.433 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 351920, 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.434 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.435 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.436 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:19.437 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:19.438 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:19.439 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.440 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.441 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.442 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.443 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.444 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:37:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:37:19.445 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:19.446 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:19.447 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:37:19.447 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 10 librados: watch_flush enter 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 10 librados: watch_flush exit 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 20 client.84377.objecter shutdown clearing up homeless session... 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 10 client.84377.objecter successfully canceled tick 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 >> v1:172.21.15.146:6812/435598642 conn(0x7ff82c03e1e0 legacy=0x7ff82c0406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 10 monclient: shutdown 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 >> v1:172.21.15.78:6790/0 conn(0x7ff84c13c220 legacy=0x7ff84c1d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff84b7fe640 1 -- 172.21.15.78:0/329159965 reap_dead start 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff84b7fe640 1 -- 172.21.15.78:0/329159965 reap_dead start 2024-09-15T20:37:19.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 shutdown_connections 2024-09-15T20:37:19.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 >> 172.21.15.78:0/329159965 conn(0x7ff84c0a63c0 msgr2=0x7ff84c12ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:19.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 shutdown_connections 2024-09-15T20:37:19.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 1 -- 172.21.15.78:0/329159965 wait complete. 2024-09-15T20:37:19.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:19.411+0000 7ff8514d7640 1 librados: shutdown 2024-09-15T20:37:19.450 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:37:19.450 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T20:37:19.450 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-09-15T20:37:19.450 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T20:37:19.451 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T20:37:19.451 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T20:37:19.451 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:37:19.460 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T20:37:19.460 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-15T20:37:19.461 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-15T20:37:19.461 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T20:37:34.461 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-15T20:37:34.462 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-15T20:37:34.462 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:37:34.462 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:37:34.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient: get_monmap_and_config 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient: build_initial_monmap 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient: monmap: 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:34.590361+0000 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:34.590361+0000 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:34.590 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 1 Processor -- start 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 1 -- start start 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient: init 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient(hunting): picked mon.noname-c con 0x7f12a8134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient(hunting): picked mon.noname-a con 0x7f12a809f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient(hunting): picked mon.noname-b con 0x7f12a806e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a80a0470 con 0x7f12a809f5a0 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a809feb0 con 0x7f12a8134bb0 2024-09-15T20:37:34.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a809e4a0 con 0x7f12a806e960 2024-09-15T20:37:34.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:34.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12acac3640 10 monclient(hunting): authenticate will time out at 1515.474891s 2024-09-15T20:37:34.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12a5d9b640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f12a81399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:34.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12a6d9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f12a806e960 0x7f12a813d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43840/0 (socket says 172.21.15.78:43840) 2024-09-15T20:37:34.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.587+0000 7f12a6d9d640 1 -- 172.21.15.78:0/1463772266 learned_addr learned my addr 172.21.15.78:0/1463772266 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:34.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:34.791+0000 7f12a5d9b640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f12a81399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:35.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:35.195+0000 7f12a5d9b640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f12a81399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:36.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:35.999+0000 7f12a5d9b640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f12a81399c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 10 monclient(hunting): tick 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 1 monclient(hunting): continuing hunt 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6790/0 conn(0x7f12a806e960 legacy=0x7f12a813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.146:6789/0 conn(0x7f12a8134bb0 legacy=0x7f12a8134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f12a81399c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 10 monclient(hunting): picked mon.noname-a con 0x7f12a809f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 10 monclient(hunting): picked mon.noname-b con 0x7f12a806e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 10 monclient(hunting): picked mon.noname-c con 0x7f12a8134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:37.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 1 -- 172.21.15.78:0/1463772266 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a80a0ff0 con 0x7f12a809f5a0 2024-09-15T20:37:37.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 1 -- 172.21.15.78:0/1463772266 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f128800eec0 con 0x7f12a8134bb0 2024-09-15T20:37:37.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 1 -- 172.21.15.78:0/1463772266 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1288010160 con 0x7f12a806e960 2024-09-15T20:37:37.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f1297fff640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:37.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.587+0000 7f12a659c640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f1288005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:37.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:37.791+0000 7f12a659c640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f1288005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:38.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:38.195+0000 7f12a659c640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f1288005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:39.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:38.999+0000 7f12a659c640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f1288005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:39.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 833326611 0 0) 0x7f1288010160 con 0x7f12a806e960 2024-09-15T20:37:39.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient(hunting): _negotiate 2024-09-15T20:37:39.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:39.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:39.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient(hunting): my global_id is 84380 2024-09-15T20:37:39.491 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1290003530 con 0x7f12a806e960 2024-09-15T20:37:39.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3049970089 0 0) 0x7f1290003530 con 0x7f12a806e960 2024-09-15T20:37:39.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.146:6789/0 conn(0x7f12a8134bb0 legacy=0x7f128800c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6789/0 conn(0x7f12a809f5a0 legacy=0x7f1288005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:37:39.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient: _finish_hunting 0 2024-09-15T20:37:39.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 1 monclient: found mon.noname-b 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1288011400 con 0x7f12a806e960 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient: _finish_auth 0 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient: _check_auth_tickets 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f129c0041e0 con 0x7f12a806e960 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12acac3640 5 monclient: authenticate success, global_id 84380 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12acac3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12a559a640 10 monclient: dump: 2024-09-15T20:37:39.493 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.487+0000 7f12acac3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f129c0043f0 con 0x7f12a806e960 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12a559a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:39.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12a559a640 1 -- 172.21.15.78:0/1463772266 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f129c004840 con 0x7f12a806e960 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 4 set_mon_vals no callback set 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12a559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12a559a640 10 monclient: dump: 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.495 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.496 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.497 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 10 monclient: shutdown 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 1 -- 172.21.15.78:0/1463772266 >> v1:172.21.15.78:6790/0 conn(0x7f12a806e960 legacy=0x7f1288008eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 1 -- 172.21.15.78:0/1463772266 shutdown_connections 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 1 -- 172.21.15.78:0/1463772266 >> 172.21.15.78:0/1463772266 conn(0x7f12a80a63c0 msgr2=0x7f12a80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 1 -- 172.21.15.78:0/1463772266 shutdown_connections 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 1 -- 172.21.15.78:0/1463772266 wait complete. 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 10 monclient: build_initial_monmap 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 10 monclient: monmap: 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:39.498 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 1 librados: starting msgr at 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.491+0000 7f12acac3640 1 librados: starting objecter 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 1 Processor -- start 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 1 -- start start 2024-09-15T20:37:39.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 1 librados: setting wanted keys 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 1 librados: calling monclient init 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 10 monclient: init 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 10 monclient(hunting): picked mon.noname-a con 0x7f12a806e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 10 monclient(hunting): picked mon.noname-c con 0x7f12a809f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12a659c640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f12a806e960 legacy=0x7f12a81339b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 10 monclient(hunting): picked mon.noname-b con 0x7f12a8134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a8073240 con 0x7f12a806e960 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a813eea0 con 0x7f12a8134bb0 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12a813f180 con 0x7f12a809f5a0 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12a5d9b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f12a809f5a0 0x7f12a81d1bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43854/0 (socket says 172.21.15.78:43854) 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12a5d9b640 1 -- 172.21.15.78:0/3015642389 learned_addr learned my addr 172.21.15.78:0/3015642389 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:39.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:39.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f12acac3640 10 monclient(hunting): authenticate will time out at 1520.381514s 2024-09-15T20:37:39.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f1296ffd640 10 client.?.objecter ms_handle_connect 0x7f12a809f5a0 2024-09-15T20:37:39.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f1296ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:39.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f1296ffd640 10 client.?.objecter ms_handle_connect 0x7f12a8134bb0 2024-09-15T20:37:39.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.495+0000 7f1296ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 960048837 0 0) 0x7f12a813f180 con 0x7f12a809f5a0 2024-09-15T20:37:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:39.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): my global_id is 84383 2024-09-15T20:37:39.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f128800f850 con 0x7f12a809f5a0 2024-09-15T20:37:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1027495937 0 0) 0x7f128800f850 con 0x7f12a809f5a0 2024-09-15T20:37:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f12a813f180 con 0x7f12a809f5a0 2024-09-15T20:37:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f129c004740 con 0x7f12a809f5a0 2024-09-15T20:37:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:39.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): dump: 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 971032749 0 0) 0x7f12a813eea0 con 0x7f12a8134bb0 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:39.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient(hunting): my global_id is 23440 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f128800f850 con 0x7f12a8134bb0 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1187799695 0 0) 0x7f12a813f180 con 0x7f12a809f5a0 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 >> v1:172.21.15.146:6789/0 conn(0x7f12a8134bb0 legacy=0x7f12a81d5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 >> v1:172.21.15.78:6789/0 conn(0x7f12a806e960 legacy=0x7f12a81339b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 monclient: found mon.c 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12a80a4710 con 0x7f12a809f5a0 2024-09-15T20:37:39.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient: _finish_auth 0 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 5 monclient: authenticate success, global_id 84383 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 10 monclient: _renew_subs 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12a813d980 con 0x7f12a809f5a0 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f129c004b90 con 0x7f12a809f5a0 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 10 client.84383.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 10 monclient: _renew_subs 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f129c004fe0 con 0x7f12a809f5a0 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f12a8142880 con 0x7f12a809f5a0 2024-09-15T20:37:39.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12a4d99640 4 set_mon_vals no callback set 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 1 librados: init done 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 monclient: dump: 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.534 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f129c005200 con 0x7f12a809f5a0 2024-09-15T20:37:39.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:39.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:39.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f12a8142c10 con 0x7f12a809f5a0 2024-09-15T20:37:39.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(263..263 src has 1..263) ==== 5364+0+0 (unknown 1454912694 0 0) 0x7f129c04b820 con 0x7f12a809f5a0 2024-09-15T20:37:39.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 10 client.84383.objecter ms_dispatch 0x7f12a8000ba0 osd_map(263..263 src has 1..263) 2024-09-15T20:37:39.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 3 client.84383.objecter handle_osd_map got epochs [263,263] > 0 2024-09-15T20:37:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 3 client.84383.objecter handle_osd_map decoding full epoch 263 2024-09-15T20:37:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f1296ffd640 20 client.84383.objecter dump_active .. 0 homeless 2024-09-15T20:37:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.527+0000 7f12a659c640 10 monclient: get_auth_request con 0x7f128803e670 auth_method 0 2024-09-15T20:37:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.531+0000 7f1296ffd640 10 client.84383.objecter ms_handle_connect 0x7f128803e670 2024-09-15T20:37:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.531+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f129c016c30 con 0x7f12a809f5a0 2024-09-15T20:37:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.535+0000 7f1296ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:39.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.535+0000 7f1296ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:37:39.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:37:39.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f12a813ec70 con 0x7f12a809f5a0 2024-09-15T20:37:39.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f1296ffd640 1 -- 172.21.15.78:0/3015642389 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 60324053) 0x7f129c016530 con 0x7f12a809f5a0 2024-09-15T20:37:39.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f1296ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:37:39.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f1296ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:39.802 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":26,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 10 librados: watch_flush enter 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 10 librados: watch_flush exit 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 20 client.84383.objecter shutdown clearing up homeless session... 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 10 client.84383.objecter successfully canceled tick 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 >> v1:172.21.15.146:6812/435598642 conn(0x7f128803e670 legacy=0x7f1288040b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 10 monclient: shutdown 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 >> v1:172.21.15.78:6790/0 conn(0x7f12a809f5a0 legacy=0x7f12a81d1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 shutdown_connections 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 >> 172.21.15.78:0/3015642389 conn(0x7f12a80a63c0 msgr2=0x7f12a80a4d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 shutdown_connections 2024-09-15T20:37:39.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 1 -- 172.21.15.78:0/3015642389 wait complete. 2024-09-15T20:37:39.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.799+0000 7f12acac3640 1 librados: shutdown 2024-09-15T20:37:39.820 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:37:39.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T20:37:39.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:37:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.939+0000 7fd1b2958640 10 monclient: get_monmap_and_config 2024-09-15T20:37:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.939+0000 7fd1b2958640 10 monclient: build_initial_monmap 2024-09-15T20:37:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.939+0000 7fd1b2958640 10 monclient: monmap: 2024-09-15T20:37:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:39.943 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:39.944000+0000 2024-09-15T20:37:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:39.944000+0000 2024-09-15T20:37:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:39.944 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:39.944 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.939+0000 7fd1b2958640 1 Processor -- start 2024-09-15T20:37:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.939+0000 7fd1b2958640 1 -- start start 2024-09-15T20:37:39.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: init 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): picked mon.noname-a con 0x7fd1ac0a0150 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): picked mon.noname-b con 0x7fd1ac137cd0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): picked mon.noname-c con 0x7fd1ac13e2b0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1ac072fd0 con 0x7fd1ac0a0150 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1ac0742f0 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1ac0745d0 con 0x7fd1ac137cd0 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): authenticate will time out at 1520.828756s 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1abfff640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fd1ac0a0150 legacy=0x7fd1ac0a0530 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1ab7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd1ac137cd0 0x7fd1ac13cb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43866/0 (socket says 172.21.15.78:43866) 2024-09-15T20:37:39.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1ab7fe640 1 -- 172.21.15.78:0/3595894638 learned_addr learned my addr 172.21.15.78:0/3595894638 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2587055917 0 0) 0x7fd1ac0745d0 con 0x7fd1ac137cd0 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient(hunting): _negotiate 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient(hunting): my global_id is 84386 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd198003530 con 0x7fd1ac137cd0 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1052133985 0 0) 0x7fd198003530 con 0x7fd1ac137cd0 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 >> v1:172.21.15.146:6789/0 conn(0x7fd1ac13e2b0 legacy=0x7fd1ac1406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 >> v1:172.21.15.78:6789/0 conn(0x7fd1ac0a0150 legacy=0x7fd1ac0a0530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: _finish_hunting 0 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 monclient: found mon.noname-b 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:39.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac0a0c40 con 0x7fd1ac137cd0 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: _finish_auth 0 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: _check_auth_tickets 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd194001a20 con 0x7fd1ac137cd0 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: dump: 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 5 monclient: authenticate success, global_id 84386 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd194001c40 con 0x7fd1ac137cd0 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 4 set_mon_vals no callback set 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 1 -- 172.21.15.78:0/3595894638 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd1940020d0 con 0x7fd1ac137cd0 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1aaffd640 10 monclient: dump: 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: shutdown 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:39.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- 172.21.15.78:0/3595894638 >> v1:172.21.15.78:6790/0 conn(0x7fd1ac137cd0 legacy=0x7fd1ac13cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- 172.21.15.78:0/3595894638 shutdown_connections 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- 172.21.15.78:0/3595894638 >> 172.21.15.78:0/3595894638 conn(0x7fd1ac0a70e0 msgr2=0x7fd1ac0a3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- 172.21.15.78:0/3595894638 shutdown_connections 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- 172.21.15.78:0/3595894638 wait complete. 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: build_initial_monmap 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: monmap: 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:39.957 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 librados: starting msgr at 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 librados: starting objecter 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 Processor -- start 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- start start 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 librados: setting wanted keys 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 librados: calling monclient init 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: init 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): picked mon.noname-b con 0x7fd1ac0a0150 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:39.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): picked mon.noname-a con 0x7fd1ac137cd0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): picked mon.noname-c con 0x7fd1ac13e2b0 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1ac074e20 con 0x7fd1ac137cd0 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1ac1390b0 con 0x7fd1ac0a0150 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1ac139390 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b2958640 10 monclient(hunting): authenticate will time out at 1520.831900s 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1ab7fe640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fd1ac137cd0 legacy=0x7fd1ac09dca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b0ef5640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd1ac13e2b0 0x7fd1ac1d4ec0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43880/0 (socket says 172.21.15.78:43880) 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.943+0000 7fd1b0ef5640 1 -- 172.21.15.78:0/2654601426 learned_addr learned my addr 172.21.15.78:0/2654601426 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1ac13e2b0 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:39.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1562452883 0 0) 0x7fd1ac139390 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): my global_id is 84389 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd18c004ec0 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1ac0a0150 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2001289554 0 0) 0x7fd18c004ec0 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1ac139390 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd19c0019c0 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:39.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): dump: 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:39.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 816784358 0 0) 0x7fd1ac1390b0 con 0x7fd1ac0a0150 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): my global_id is 23446 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd18c004ec0 con 0x7fd1ac0a0150 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1366453319 0 0) 0x7fd1ac139390 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 >> v1:172.21.15.146:6789/0 conn(0x7fd1ac0a0150 legacy=0x7fd1ac09d590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 >> v1:172.21.15.78:6789/0 conn(0x7fd1ac137cd0 legacy=0x7fd1ac09dca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 monclient: found mon.c 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1ac139670 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: _finish_auth 0 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 5 monclient: authenticate success, global_id 84389 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 10 monclient: _renew_subs 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1ac13da70 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 10 client.84389.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 10 monclient: _renew_subs 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1ac09ee60 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 1 librados: init done 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd19c001c80 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd19c003a70 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: dump: 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1aa7fc640 4 set_mon_vals no callback set 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd1900065d0 con 0x7fd190002870 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 client.84389.objecter ms_handle_connect 0x7fd190002870 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 client.84389.objecter resend_mon_ops 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1770172408 0 0) 0x7fd1900065d0 con 0x7fd190002870 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient(hunting): my global_id is 84392 2024-09-15T20:37:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1ac139390 con 0x7fd190002870 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd19c003c10 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(264..264 src has 1..264) ==== 5743+0+0 (unknown 1243179579 0 0) 0x7fd19c04b820 con 0x7fd1ac13e2b0 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 client.84389.objecter ms_dispatch 0x7fd1ac000ba0 osd_map(264..264 src has 1..264) 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 3 client.84389.objecter handle_osd_map got epochs [264,264] > 0 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 3 client.84389.objecter handle_osd_map decoding full epoch 264 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 20 client.84389.objecter dump_active .. 0 homeless 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1472263610 0 0) 0x7fd1ac139390 con 0x7fd190002870 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1900065d0 con 0x7fd190002870 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd1a4004030 con 0x7fd190002870 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: dump: 2024-09-15T20:37:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2872343674 0 0) 0x7fd1900065d0 con 0x7fd190002870 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1ab7fe640 10 monclient: get_auth_request con 0x7fd18c03e110 auth_method 0 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd19000a170 con 0x7fd190002870 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd1a4006370 con 0x7fd190002870 2024-09-15T20:37:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 >> v1:172.21.15.78:6790/0 conn(0x7fd190002870 legacy=0x7fd190004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 >> v1:172.21.15.78:6790/0 conn(0x7fd190002870 legacy=0x7fd190004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.947+0000 7fd1a8ff9640 10 client.84389.objecter ms_handle_connect 0x7fd18c03e110 2024-09-15T20:37:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:37:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd190009e60 con 0x7fd190009120 2024-09-15T20:37:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 client.84389.objecter ms_handle_connect 0x7fd190009120 2024-09-15T20:37:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 client.84389.objecter resend_mon_ops 2024-09-15T20:37:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1778253774 0 0) 0x7fd190009e60 con 0x7fd190009120 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient(hunting): my global_id is 84395 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1900065d0 con 0x7fd190009120 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1898733540 0 0) 0x7fd1900065d0 con 0x7fd190009120 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd190009e60 con 0x7fd190009120 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd1a40056c0 con 0x7fd190009120 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient: dump: 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4126840691 0 0) 0x7fd190009e60 con 0x7fd190009120 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd19000c400 con 0x7fd190009120 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4127 (unknown 0 0 2386723734) 0x7fd1a4006620 con 0x7fd190009120 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 >> v1:172.21.15.78:6790/0 conn(0x7fd190009120 legacy=0x7fd1900095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1a8ff9640 1 -- 172.21.15.78:0/2654601426 >> v1:172.21.15.78:6790/0 conn(0x7fd190009120 legacy=0x7fd1900095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 26, 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 131642, 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:37:39.972 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:39.973 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:39.974 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:39.975 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:39.976 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:39.977 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:39.978 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.979 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:37:39.980 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:37:39.981 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:37:39.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:39.983 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:39.983 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:39.983 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:37:39.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:39.983 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:39.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:39.983 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:37:39.983 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 10 librados: watch_flush enter 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 10 librados: watch_flush exit 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 20 client.84389.objecter shutdown clearing up homeless session... 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 10 client.84389.objecter successfully canceled tick 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 >> v1:172.21.15.146:6812/435598642 conn(0x7fd18c03e110 legacy=0x7fd18c0405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 10 monclient: shutdown 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 >> v1:172.21.15.78:6790/0 conn(0x7fd1ac13e2b0 legacy=0x7fd1ac1d4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b0ef5640 1 -- 172.21.15.78:0/2654601426 reap_dead start 2024-09-15T20:37:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b0ef5640 1 -- 172.21.15.78:0/2654601426 reap_dead start 2024-09-15T20:37:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 shutdown_connections 2024-09-15T20:37:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 >> 172.21.15.78:0/2654601426 conn(0x7fd1ac0a70e0 msgr2=0x7fd1ac1406a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 shutdown_connections 2024-09-15T20:37:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 1 -- 172.21.15.78:0/2654601426 wait complete. 2024-09-15T20:37:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:39.955+0000 7fd1b2958640 1 librados: shutdown 2024-09-15T20:37:39.986 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:37:39.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T20:37:39.986 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:37:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient: get_monmap_and_config 2024-09-15T20:37:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient: build_initial_monmap 2024-09-15T20:37:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient: monmap: 2024-09-15T20:37:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:37:40.101784+0000 2024-09-15T20:37:40.101 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:37:40.101784+0000 2024-09-15T20:37:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:37:40.102 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:37:40.102 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 1 Processor -- start 2024-09-15T20:37:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 1 -- start start 2024-09-15T20:37:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient: init 2024-09-15T20:37:40.109 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient(hunting): picked mon.noname-a con 0x7f65fc135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient(hunting): picked mon.noname-c con 0x7f65fc0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient(hunting): picked mon.noname-b con 0x7f65fc13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc0745e0 con 0x7f65fc135b60 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc06ecf0 con 0x7f65fc0a24e0 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc06efd0 con 0x7f65fc13c220 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient(hunting): authenticate will time out at 1520.986619s 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f66017fc640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f65fc135b60 legacy=0x7f65fc135f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6601ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f65fc13c220 0x7f65fc13e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43908/0 (socket says 172.21.15.78:43908) 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6601ffd640 1 -- 172.21.15.78:0/2548557092 learned_addr learned my addr 172.21.15.78:0/2548557092 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:40.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2186313827 0 0) 0x7f65fc06efd0 con 0x7f65fc13c220 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient(hunting): _negotiate 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient(hunting): my global_id is 84398 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e8003530 con 0x7f65fc13c220 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1839615578 0 0) 0x7f65e8003530 con 0x7f65fc13c220 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 >> v1:172.21.15.146:6789/0 conn(0x7f65fc0a24e0 legacy=0x7f65fc13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 >> v1:172.21.15.78:6789/0 conn(0x7f65fc135b60 legacy=0x7f65fc135f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: _finish_hunting 0 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 monclient: found mon.noname-b 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:37:40.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc136650 con 0x7f65fc13c220 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: _finish_auth 0 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: _check_auth_tickets 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65f4001a20 con 0x7f65fc13c220 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: dump: 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:40.112 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 5 monclient: authenticate success, global_id 84398 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65f4001c40 con 0x7f65fc13c220 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 1 -- 172.21.15.78:0/2548557092 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65f4002050 con 0x7f65fc13c220 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f65f3fff640 10 monclient: dump: 2024-09-15T20:37:40.113 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient: get_monmap_and_config success 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 4 set_mon_vals no callback set 2024-09-15T20:37:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 10 monclient: shutdown 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 1 -- 172.21.15.78:0/2548557092 >> v1:172.21.15.78:6790/0 conn(0x7f65fc13c220 legacy=0x7f65fc13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 1 -- 172.21.15.78:0/2548557092 shutdown_connections 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.099+0000 7f6603a60640 1 -- 172.21.15.78:0/2548557092 >> 172.21.15.78:0/2548557092 conn(0x7f65fc0a63c0 msgr2=0x7f65fc0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- 172.21.15.78:0/2548557092 shutdown_connections 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- 172.21.15.78:0/2548557092 wait complete. 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: build_initial_monmap 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: monmap: 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:37:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 librados: starting msgr at 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 librados: starting objecter 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 Processor -- start 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- start start 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 librados: setting wanted keys 2024-09-15T20:37:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 librados: calling monclient init 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: init 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: _reopen_session rank -1 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient(hunting): picked mon.noname-a con 0x7f65fc0a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient(hunting): picked mon.noname-c con 0x7f65fc135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient(hunting): picked mon.noname-b con 0x7f65fc13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc136f90 con 0x7f65fc0a24e0 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc074d70 con 0x7f65fc13c220 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65fc0736f0 con 0x7f65fc135b60 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient(hunting): _renew_subs 2024-09-15T20:37:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f66017fc640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f65fc0a24e0 legacy=0x7f65fc071f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient(hunting): authenticate will time out at 1520.989474s 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6600ffb640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f65fc135b60 0x7f65fc0726a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:43912/0 (socket says 172.21.15.78:43912) 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6600ffb640 1 -- 172.21.15.78:0/428976135 learned_addr learned my addr 172.21.15.78:0/428976135 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 client.?.objecter ms_handle_connect 0x7f65fc135b60 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 476260954 0 0) 0x7f65fc0736f0 con 0x7f65fc135b60 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): my global_id is 84401 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e0004ef0 con 0x7f65fc135b60 2024-09-15T20:37:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 client.?.objecter ms_handle_connect 0x7f65fc13c220 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 639304115 0 0) 0x7f65e0004ef0 con 0x7f65fc135b60 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65fc0736f0 con 0x7f65fc135b60 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65ec001a00 con 0x7f65fc135b60 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): dump: 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 405106543 0 0) 0x7f65fc074d70 con 0x7f65fc13c220 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): my global_id is 23452 2024-09-15T20:37:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65e0004ef0 con 0x7f65fc13c220 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1353215700 0 0) 0x7f65fc0736f0 con 0x7f65fc135b60 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 >> v1:172.21.15.146:6789/0 conn(0x7f65fc13c220 legacy=0x7f65fc1d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 >> v1:172.21.15.78:6789/0 conn(0x7f65fc0a24e0 legacy=0x7f65fc071f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 monclient: found mon.c 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc138790 con 0x7f65fc135b60 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient: _finish_auth 0 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:37:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 5 monclient: authenticate success, global_id 84401 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: _renew_subs 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc137ef0 con 0x7f65fc135b60 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 client.84401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: _renew_subs 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f65fc072db0 con 0x7f65fc135b60 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 librados: init done 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f65ec001cc0 con 0x7f65fc135b60 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:37:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65ec003a70 con 0x7f65fc135b60 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient: dump: 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f37fe640 4 set_mon_vals no callback set 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f65dc0065d0 con 0x7f65dc002870 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f65ec003c10 con 0x7f65fc135b60 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 client.84401.objecter ms_handle_connect 0x7f65dc002870 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 client.84401.objecter resend_mon_ops 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(264..264 src has 1..264) ==== 5743+0+0 (unknown 1243179579 0 0) 0x7f65ec04b9e0 con 0x7f65fc135b60 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 client.84401.objecter ms_dispatch 0x7f65fc000ba0 osd_map(264..264 src has 1..264) 2024-09-15T20:37:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 3 client.84401.objecter handle_osd_map got epochs [264,264] > 0 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 3 client.84401.objecter handle_osd_map decoding full epoch 264 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 20 client.84401.objecter dump_active .. 0 homeless 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252407945 0 0) 0x7f65dc0065d0 con 0x7f65dc002870 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 10 monclient(hunting): my global_id is 23455 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65fc0736f0 con 0x7f65dc002870 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.103+0000 7f6601ffd640 10 monclient: get_auth_request con 0x7f65e003e100 auth_method 0 2024-09-15T20:37:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3274504384 0 0) 0x7f65fc0736f0 con 0x7f65dc002870 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65dc0065d0 con 0x7f65dc002870 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65f8003fb0 con 0x7f65dc002870 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 10 monclient: dump: 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 10 client.84401.objecter ms_handle_connect 0x7f65e003e100 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1521650107 0 0) 0x7f65dc0065d0 con 0x7f65dc002870 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f65dc00a170 con 0x7f65dc002870 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f65f8006350 con 0x7f65dc002870 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:37:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:37:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 >> v1:172.21.15.146:6789/0 conn(0x7f65dc002870 legacy=0x7f65dc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.107+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 >> v1:172.21.15.146:6789/0 conn(0x7f65dc002870 legacy=0x7f65dc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.111+0000 7f6603a60640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.111+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f65dc009e60 con 0x7f65dc009120 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.111+0000 7f65f1ffb640 10 client.84401.objecter ms_handle_connect 0x7f65dc009120 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.111+0000 7f65f1ffb640 10 client.84401.objecter resend_mon_ops 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 318663674 0 0) 0x7f65dc009e60 con 0x7f65dc009120 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient(hunting): my global_id is 23458 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65dc0065d0 con 0x7f65dc009120 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4049190687 0 0) 0x7f65dc0065d0 con 0x7f65dc009120 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f65dc009e60 con 0x7f65dc009120 2024-09-15T20:37:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f65f8004730 con 0x7f65dc009120 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient: dump: 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:37:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2207756295 0 0) 0x7f65dc009e60 con 0x7f65dc009120 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f65dc00c400 con 0x7f65dc009120 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 1399948582) 0x7f65f8005ba0 con 0x7f65dc009120 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 >> v1:172.21.15.146:6789/0 conn(0x7f65dc009120 legacy=0x7f65dc0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f65f1ffb640 1 -- 172.21.15.78:0/428976135 >> v1:172.21.15.146:6789/0 conn(0x7f65dc009120 legacy=0x7f65dc0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:40.132 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 26, 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 372623, 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:40.133 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:40.134 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:37:40.135 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:37:40.136 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:37:40.137 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:40.138 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:40.139 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:37:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:40.140 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:37:40.141 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:40.142 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:40.143 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:37:40.144 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:37:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:37:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:37:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:37:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:37:40.145 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:37:40.145 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:37:40.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 10 librados: watch_flush enter 2024-09-15T20:37:40.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 10 librados: watch_flush exit 2024-09-15T20:37:40.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 20 client.84401.objecter shutdown clearing up homeless session... 2024-09-15T20:37:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 10 client.84401.objecter successfully canceled tick 2024-09-15T20:37:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 >> v1:172.21.15.146:6812/435598642 conn(0x7f65e003e100 legacy=0x7f65e00405c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 10 monclient: shutdown 2024-09-15T20:37:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:37:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 >> v1:172.21.15.78:6790/0 conn(0x7f65fc135b60 legacy=0x7f65fc0726a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:37:40.147 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6601ffd640 1 -- 172.21.15.78:0/428976135 reap_dead start 2024-09-15T20:37:40.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6601ffd640 1 -- 172.21.15.78:0/428976135 reap_dead start 2024-09-15T20:37:40.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 shutdown_connections 2024-09-15T20:37:40.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 >> 172.21.15.78:0/428976135 conn(0x7f65fc0a63c0 msgr2=0x7f65fc12eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:37:40.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 shutdown_connections 2024-09-15T20:37:40.148 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 1 -- 172.21.15.78:0/428976135 wait complete. 2024-09-15T20:37:40.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:37:40.115+0000 7f6603a60640 1 librados: shutdown 2024-09-15T20:37:40.149 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:37:40.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:38:01.657 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.RoundTripWriteFullPP2::163:head 2024-09-15T20:38:01.657 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:5d165639:::164:head 2024-09-15T20:38:01.657 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f43765fc:::165:head 2024-09-15T20:38:01.657 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b4c720e9:::166:head 2024-09-15T20:38:01.657 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:e694b040:::167:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:afa38db2:::168:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:77ba9f53:::169:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:87495034:::170:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:7c96bf0e:::171:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:dbe346cc:::172:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:e943ec24:::173:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f97a9c0c:::174:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:6f26e74d:::175:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:4f95e106:::176:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:0e6f2f8f:::177:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:05db05f1:::178:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:38a78d66:::179:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:d095610b:::180:head 2024-09-15T20:38:01.658 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:a1a9d709:::181:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:1e5d39db:::182:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:f7df4fb9:::183:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:03a7f161:::184:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:ba70721e:::185:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:28e5662d:::186:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:973d52de:::187:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:4303eb1c:::188:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:b990b48e:::189:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:29b8165b:::190:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:3547f197:::191:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:7e260936:::192:head 2024-09-15T20:38:01.659 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:1abec7b1:::193:head 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:10fdda93:::194:head 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:15817eea:::195:head 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:770bab57:::196:head 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:ed9e13e7:::197:head 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:71471a8f:::198:head 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: checking for 270:10fb1d02:::199:head 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetWrite (8987 ms) 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HitSetTrim 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,0 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: first is 1726432616 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,0 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,0 2024-09-15T20:38:01.660 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,1726432621,1726432622,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,1726432621,1726432622,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,1726432621,1726432622,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,1726432621,1726432622,1726432624,1726432625,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,1726432621,1726432622,1726432624,1726432625,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432616,1726432618,1726432619,1726432621,1726432622,1726432624,1726432625,0 2024-09-15T20:38:01.661 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726432619,1726432621,1726432622,1726432624,1726432625,1726432627,1726432628,0 2024-09-15T20:38:01.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: first now 1726432619, trimmed 2024-09-15T20:38:01.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HitSetTrim (21013 ms) 2024-09-15T20:38:01.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PromoteOn2ndRead 2024-09-15T20:38:01.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: foo0 2024-09-15T20:38:01.662 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-15T20:38:01.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PromoteOn2ndRead (33998 ms) 2024-09-15T20:38:01.663 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ProxyRead 2024-09-15T20:38:23.633 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.RoundTripWriteFullPP2 (23045 ms) 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPP 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPP (7024 ms) 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.SimpleStatPPNS 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.SimpleStatPPNS (7016 ms) 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.StatRemovePP 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.StatRemovePP (6985 ms) 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.ExecuteClassPP 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.ExecuteClassPP (7021 ms) 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.OmapPP 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.OmapPP (6987 ms) 2024-09-15T20:38:23.634 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ RUN ] LibRadosAioEC.MultiWritePP 2024-09-15T20:38:23.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ OK ] LibRadosAioEC.MultiWritePP (6994 ms) 2024-09-15T20:38:23.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] 20 tests from LibRadosAioEC (173025 ms total) 2024-09-15T20:38:23.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: 2024-09-15T20:38:23.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [----------] Global test environment tear-down 2024-09-15T20:38:23.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [==========] 57 tests from 4 test suites ran. (335044 ms total) 2024-09-15T20:38:23.635 INFO:tasks.workunit.client.0.smithi146.stdout: api_aio_pp: [ PASSED ] 57 tests. 2024-09-15T20:39:10.150 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T20:39:10.151 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T20:39:10.151 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T20:39:10.154 INFO:tasks.ceph.mon.a:Started 2024-09-15T20:39:10.154 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-09-15T20:39:10.154 INFO:tasks.ceph.mon.c:Sent signal 19 2024-09-15T20:39:10.155 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T20:39:25.156 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-09-15T20:39:25.156 INFO:tasks.ceph.mon.c:Sent signal 18 2024-09-15T20:39:25.157 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:39:25.157 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:39:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient: get_monmap_and_config 2024-09-15T20:39:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient: build_initial_monmap 2024-09-15T20:39:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient: monmap: 2024-09-15T20:39:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:25.294307+0000 2024-09-15T20:39:25.293 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:25.294307+0000 2024-09-15T20:39:25.294 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:25.294 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.294 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:25.294 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:25.294 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:25.294 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 1 Processor -- start 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 1 -- start start 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient: init 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient(hunting): picked mon.noname-c con 0x7fa3bc0a0ff0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient(hunting): picked mon.noname-b con 0x7fa3bc073800 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient(hunting): picked mon.noname-a con 0x7fa3bc0a1ae0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3bc072df0 con 0x7fa3bc0a1ae0 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3bc074110 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3bc0743f0 con 0x7fa3bc073800 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:25.301 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient(hunting): authenticate will time out at 1626.178730s 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c0a48640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa3bc0a1ae0 0x7fa3bc140400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49352/0 (socket says 172.21.15.78:49352) 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3bb7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa3bc073800 0x7fa3bc13ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51056/0 (socket says 172.21.15.78:51056) 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c0a48640 1 -- 172.21.15.78:0/1131598910 learned_addr learned my addr 172.21.15.78:0/1131598910 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2151577381 0 0) 0x7fa3bc0743f0 con 0x7fa3bc073800 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient(hunting): my global_id is 84440 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3a4003530 con 0x7fa3bc073800 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3336568149 0 0) 0x7fa3bc072df0 con 0x7fa3bc0a1ae0 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.302 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient(hunting): my global_id is 104097 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3bc0743f0 con 0x7fa3bc0a1ae0 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1710222821 0 0) 0x7fa3a4003530 con 0x7fa3bc073800 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 >> v1:172.21.15.146:6789/0 conn(0x7fa3bc0a0ff0 legacy=0x7fa3bc0a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 >> v1:172.21.15.78:6789/0 conn(0x7fa3bc0a1ae0 legacy=0x7fa3bc140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: _finish_hunting 0 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 monclient: found mon.noname-b 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3bc074eb0 con 0x7fa3bc073800 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: _finish_auth 0 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: _check_auth_tickets 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa3ac0016a0 con 0x7fa3bc073800 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 5 monclient: authenticate success, global_id 84440 2024-09-15T20:39:25.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: dump: 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:25.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3ac0018c0 con 0x7fa3bc073800 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 1 -- 172.21.15.78:0/1131598910 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa3ac001c90 con 0x7fa3bc073800 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3c24ab640 4 set_mon_vals no callback set 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.290+0000 7fa3baffd640 10 monclient: dump: 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.305 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: shutdown 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1131598910 >> v1:172.21.15.78:6790/0 conn(0x7fa3bc073800 legacy=0x7fa3bc13ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1131598910 shutdown_connections 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1131598910 >> 172.21.15.78:0/1131598910 conn(0x7fa3bc0a63c0 msgr2=0x7fa3bc0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1131598910 shutdown_connections 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1131598910 wait complete. 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: build_initial_monmap 2024-09-15T20:39:25.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: monmap: 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 librados: starting msgr at 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 librados: starting objecter 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 Processor -- start 2024-09-15T20:39:25.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- start start 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 librados: setting wanted keys 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 librados: calling monclient init 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: init 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient(hunting): picked mon.noname-a con 0x7fa3bc073800 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient(hunting): picked mon.noname-c con 0x7fa3bc0a0ff0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient(hunting): picked mon.noname-b con 0x7fa3bc0a1ae0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3bc11fc10 con 0x7fa3bc073800 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3bc11d400 con 0x7fa3bc0a1ae0 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa3bc09dd70 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:25.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient(hunting): authenticate will time out at 1626.181795s 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3bb7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa3bc0a0ff0 0x7fa3bc11ccf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51064/0 (socket says 172.21.15.78:51064) 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3bb7fe640 1 -- 172.21.15.78:0/1544647281 learned_addr learned my addr 172.21.15.78:0/1544647281 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa3bc0a0ff0 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa3bc073800 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2988184362 0 0) 0x7fa3bc09dd70 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): my global_id is 84443 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3a8004ec0 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1382372829 0 0) 0x7fa3bc11fc10 con 0x7fa3bc073800 2024-09-15T20:39:25.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): my global_id is 104100 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3bc09dd70 con 0x7fa3bc073800 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 client.?.objecter ms_handle_connect 0x7fa3bc0a1ae0 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 418400547 0 0) 0x7fa3a8004ec0 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3bc11fc10 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa3ac001c10 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): dump: 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.310 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2846166250 0 0) 0x7fa3bc09dd70 con 0x7fa3bc073800 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa3a8004ec0 con 0x7fa3bc073800 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa3b0001900 con 0x7fa3bc073800 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): dump: 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.311 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3184976077 0 0) 0x7fa3bc11d400 con 0x7fa3bc0a1ae0 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient(hunting): my global_id is 23500 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa3bc09dd70 con 0x7fa3bc0a1ae0 2024-09-15T20:39:25.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 127511057 0 0) 0x7fa3bc11fc10 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 >> v1:172.21.15.146:6789/0 conn(0x7fa3bc0a1ae0 legacy=0x7fa3bc1d5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 >> v1:172.21.15.78:6789/0 conn(0x7fa3bc073800 legacy=0x7fa3bc11c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 monclient: found mon.c 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3bc09d180 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient: _finish_auth 0 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 5 monclient: authenticate success, global_id 84443 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: _renew_subs 2024-09-15T20:39:25.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa3bc138760 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 client.84443.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: _renew_subs 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa3bc11f930 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 librados: init done 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa3ac001180 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa3ac004ce0 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3ba7fc640 4 set_mon_vals no callback set 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 monclient: dump: 2024-09-15T20:39:25.314 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa39c005090 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fa3ac010d50 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(339..339 src has 1..339) ==== 5346+0+0 (unknown 691184420 0 0) 0x7fa3ac04b7d0 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 10 client.84443.objecter ms_dispatch 0x7fa3bc000ba0 osd_map(339..339 src has 1..339) 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 3 client.84443.objecter handle_osd_map got epochs [339,339] > 0 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.294+0000 7fa3b8ff9640 3 client.84443.objecter handle_osd_map decoding full epoch 339 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.298+0000 7fa3b8ff9640 20 client.84443.objecter dump_active .. 0 homeless 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.298+0000 7fa3bbfff640 10 monclient: get_auth_request con 0x7fa3a803e210 auth_method 0 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.298+0000 7fa3b8ff9640 10 client.84443.objecter ms_handle_connect 0x7fa3a803e210 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.302+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fa3ac016b30 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.302+0000 7fa3b8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:25.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.302+0000 7fa3b8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.534+0000 7fa3c24ab640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:39:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.534+0000 7fa3c24ab640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.534+0000 7fa3c24ab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.534+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa39c005380 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.534+0000 7fa3b8ff9640 1 -- 172.21.15.78:0/1544647281 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2882499584) 0x7fa3ac016430 con 0x7fa3bc0a0ff0 2024-09-15T20:39:25.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.534+0000 7fa3b8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:25.539 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.534+0000 7fa3b8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:25.539 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:39:25.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 10 librados: watch_flush enter 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 10 librados: watch_flush exit 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 20 client.84443.objecter shutdown clearing up homeless session... 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 10 client.84443.objecter successfully canceled tick 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 >> v1:172.21.15.146:6812/435598642 conn(0x7fa3a803e210 legacy=0x7fa3a80406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 10 monclient: shutdown 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 >> v1:172.21.15.78:6790/0 conn(0x7fa3bc0a0ff0 legacy=0x7fa3bc11ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 shutdown_connections 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 >> 172.21.15.78:0/1544647281 conn(0x7fa3bc0a63c0 msgr2=0x7fa3bc13aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 shutdown_connections 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 1 -- 172.21.15.78:0/1544647281 wait complete. 2024-09-15T20:39:25.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.538+0000 7fa3c24ab640 1 librados: shutdown 2024-09-15T20:39:25.557 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:39:25.557 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:39:25.557 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:39:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient: get_monmap_and_config 2024-09-15T20:39:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient: build_initial_monmap 2024-09-15T20:39:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient: monmap: 2024-09-15T20:39:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.677 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:25.678590+0000 2024-09-15T20:39:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:25.678590+0000 2024-09-15T20:39:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:25.678 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:25.678 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 1 Processor -- start 2024-09-15T20:39:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 1 -- start start 2024-09-15T20:39:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient: init 2024-09-15T20:39:25.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient(hunting): picked mon.noname-c con 0x7fd9900a0ff0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient(hunting): picked mon.noname-b con 0x7fd990073800 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient(hunting): picked mon.noname-a con 0x7fd9900a1ae0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd990072df0 con 0x7fd9900a1ae0 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd990074110 con 0x7fd9900a0ff0 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9900743f0 con 0x7fd990073800 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 10 monclient(hunting): authenticate will time out at 1626.563076s 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98ffff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd990073800 0x7fd99013ccd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51078/0 (socket says 172.21.15.78:51078) 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98ffff640 1 -- 172.21.15.78:0/738633764 learned_addr learned my addr 172.21.15.78:0/738633764 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2135799809 0 0) 0x7fd990072df0 con 0x7fd9900a1ae0 2024-09-15T20:39:25.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient(hunting): my global_id is 104103 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd97c003530 con 0x7fd9900a1ae0 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 950153690 0 0) 0x7fd9900743f0 con 0x7fd990073800 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient(hunting): my global_id is 84446 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd990072df0 con 0x7fd990073800 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1799706576 0 0) 0x7fd990072df0 con 0x7fd990073800 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 >> v1:172.21.15.146:6789/0 conn(0x7fd9900a0ff0 legacy=0x7fd9900a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 >> v1:172.21.15.78:6789/0 conn(0x7fd9900a1ae0 legacy=0x7fd990140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 monclient: found mon.noname-b 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd990074eb0 con 0x7fd990073800 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 5 monclient: authenticate success, global_id 84446 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd978001680 con 0x7fd990073800 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:25.688 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd98f7fe640 10 monclient: dump: 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.674+0000 7fd996ee9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd978001820 con 0x7fd990073800 2024-09-15T20:39:25.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98f7fe640 1 -- 172.21.15.78:0/738633764 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd978001c30 con 0x7fd990073800 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 4 set_mon_vals no callback set 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98f7fe640 10 monclient: dump: 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.690 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: shutdown 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- 172.21.15.78:0/738633764 >> v1:172.21.15.78:6790/0 conn(0x7fd990073800 legacy=0x7fd99013ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- 172.21.15.78:0/738633764 shutdown_connections 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- 172.21.15.78:0/738633764 >> 172.21.15.78:0/738633764 conn(0x7fd9900a63c0 msgr2=0x7fd9900a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- 172.21.15.78:0/738633764 shutdown_connections 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- 172.21.15.78:0/738633764 wait complete. 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: build_initial_monmap 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: monmap: 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:25.691 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 librados: starting msgr at 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 librados: starting objecter 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 Processor -- start 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- start start 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 librados: setting wanted keys 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 librados: calling monclient init 2024-09-15T20:39:25.692 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: init 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient(hunting): picked mon.noname-c con 0x7fd990073800 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient(hunting): picked mon.noname-a con 0x7fd9900a0ff0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient(hunting): picked mon.noname-b con 0x7fd9900a1ae0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd990141910 con 0x7fd9900a0ff0 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd99013ac80 con 0x7fd9900a1ae0 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd99009d180 con 0x7fd990073800 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient(hunting): authenticate will time out at 1626.566313s 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd994c85640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd990073800 0x7fd990141200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51088/0 (socket says 172.21.15.78:51088) 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd994c85640 1 -- 172.21.15.78:0/3683403119 learned_addr learned my addr 172.21.15.78:0/3683403119 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 client.?.objecter ms_handle_connect 0x7fd990073800 2024-09-15T20:39:25.693 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 client.?.objecter ms_handle_connect 0x7fd9900a0ff0 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4106467348 0 0) 0x7fd99009d180 con 0x7fd990073800 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): my global_id is 84449 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd970004ec0 con 0x7fd990073800 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3116795006 0 0) 0x7fd990074eb0 con 0x7fd9900a0ff0 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): my global_id is 104106 2024-09-15T20:39:25.694 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd99009d180 con 0x7fd9900a0ff0 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 client.?.objecter ms_handle_connect 0x7fd9900a1ae0 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2565880906 0 0) 0x7fd970004ec0 con 0x7fd990073800 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd990074eb0 con 0x7fd990073800 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd988001dc0 con 0x7fd990073800 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): dump: 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.695 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2954295487 0 0) 0x7fd99009d180 con 0x7fd9900a0ff0 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd970004ec0 con 0x7fd9900a0ff0 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd978001b70 con 0x7fd9900a0ff0 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): dump: 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.696 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 64410707 0 0) 0x7fd99013ac80 con 0x7fd9900a1ae0 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient(hunting): my global_id is 23506 2024-09-15T20:39:25.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd99009d180 con 0x7fd9900a1ae0 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4079427210 0 0) 0x7fd970004ec0 con 0x7fd9900a0ff0 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 >> v1:172.21.15.78:6790/0 conn(0x7fd990073800 legacy=0x7fd990141200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 >> v1:172.21.15.146:6789/0 conn(0x7fd9900a1ae0 legacy=0x7fd9901d5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 monclient: found mon.a 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd99009d9c0 con 0x7fd9900a0ff0 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient: _finish_auth 0 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 5 monclient: authenticate success, global_id 104106 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: _renew_subs 2024-09-15T20:39:25.698 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd99013a3e0 con 0x7fd9900a0ff0 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 client.104106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: _renew_subs 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd990139650 con 0x7fd9900a0ff0 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd996ee9640 1 librados: init done 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd978004aa0 con 0x7fd9900a0ff0 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd978004df0 con 0x7fd9900a0ff0 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98effd640 4 set_mon_vals no callback set 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.678+0000 7fd98d7fa640 10 monclient: dump: 2024-09-15T20:39:25.699 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd996ee9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd9740065d0 con 0x7fd974002870 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 client.104106.objecter ms_handle_connect 0x7fd974002870 2024-09-15T20:39:25.700 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 client.104106.objecter resend_mon_ops 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1244434886 0 0) 0x7fd9740065d0 con 0x7fd974002870 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient(hunting): my global_id is 104109 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd970004ec0 con 0x7fd974002870 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1357094876 0 0) 0x7fd970004ec0 con 0x7fd974002870 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd9740065d0 con 0x7fd974002870 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd988003df0 con 0x7fd974002870 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient: dump: 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.701 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd9780010b0 con 0x7fd9900a0ff0 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(339..339 src has 1..339) ==== 5346+0+0 (unknown 691184420 0 0) 0x7fd97804b530 con 0x7fd9900a0ff0 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 client.104106.objecter ms_dispatch 0x7fd990000ba0 osd_map(339..339 src has 1..339) 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 3 client.104106.objecter handle_osd_map got epochs [339,339] > 0 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 3 client.104106.objecter handle_osd_map decoding full epoch 339 2024-09-15T20:39:25.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 20 client.104106.objecter dump_active .. 0 homeless 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3350265556 0 0) 0x7fd9740065d0 con 0x7fd974002870 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd97400a170 con 0x7fd974002870 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd995486640 10 monclient: get_auth_request con 0x7fd97003e1b0 auth_method 0 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd988005b50 con 0x7fd974002870 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 >> v1:172.21.15.78:6789/0 conn(0x7fd974002870 legacy=0x7fd974004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 >> v1:172.21.15.78:6789/0 conn(0x7fd974002870 legacy=0x7fd974004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:25.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.682+0000 7fd98d7fa640 10 client.104106.objecter ms_handle_connect 0x7fd97003e1b0 2024-09-15T20:39:25.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd996ee9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd974009e90 con 0x7fd974009150 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd98d7fa640 10 client.104106.objecter ms_handle_connect 0x7fd974009150 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd98d7fa640 10 client.104106.objecter resend_mon_ops 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4091484817 0 0) 0x7fd974009e90 con 0x7fd974009150 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd98d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd98d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd98d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd98d7fa640 10 monclient(hunting): my global_id is 104112 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.686+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd9740065d0 con 0x7fd974009150 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2038884153 0 0) 0x7fd9740065d0 con 0x7fd974009150 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd974009e90 con 0x7fd974009150 2024-09-15T20:39:25.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd9880046a0 con 0x7fd974009150 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 10 monclient: dump: 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.706 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4035245294 0 0) 0x7fd974009e90 con 0x7fd974009150 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd97400c420 con 0x7fd974009150 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3972 (unknown 0 0 4073310042) 0x7fd988004660 con 0x7fd974009150 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 >> v1:172.21.15.78:6789/0 conn(0x7fd974009150 legacy=0x7fd9740095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd98d7fa640 1 -- 172.21.15.78:0/3683403119 >> v1:172.21.15.78:6789/0 conn(0x7fd974009150 legacy=0x7fd9740095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:25.707 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 32, 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 15459, 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:39:25.708 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:39:25.709 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:39:25.710 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:39:25.711 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:25.712 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:25.713 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:25.714 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.715 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:39:25.716 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:25.717 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:39:25.718 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:39:25.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 10 librados: watch_flush enter 2024-09-15T20:39:25.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 10 librados: watch_flush exit 2024-09-15T20:39:25.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 20 client.104106.objecter shutdown clearing up homeless session... 2024-09-15T20:39:25.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 10 client.104106.objecter successfully canceled tick 2024-09-15T20:39:25.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 >> v1:172.21.15.146:6812/435598642 conn(0x7fd97003e1b0 legacy=0x7fd970040670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 10 monclient: shutdown 2024-09-15T20:39:25.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:25.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 >> v1:172.21.15.78:6789/0 conn(0x7fd9900a0ff0 legacy=0x7fd9901d1ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd995486640 1 -- 172.21.15.78:0/3683403119 reap_dead start 2024-09-15T20:39:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd995486640 1 -- 172.21.15.78:0/3683403119 reap_dead start 2024-09-15T20:39:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 shutdown_connections 2024-09-15T20:39:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 >> 172.21.15.78:0/3683403119 conn(0x7fd9900a63c0 msgr2=0x7fd9900a3140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 shutdown_connections 2024-09-15T20:39:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 1 -- 172.21.15.78:0/3683403119 wait complete. 2024-09-15T20:39:25.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.690+0000 7fd996ee9640 1 librados: shutdown 2024-09-15T20:39:25.721 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:39:25.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:39:25.722 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:39:25.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient: get_monmap_and_config 2024-09-15T20:39:25.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient: build_initial_monmap 2024-09-15T20:39:25.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient: monmap: 2024-09-15T20:39:25.850 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:25.850 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:25.838269+0000 2024-09-15T20:39:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:25.838269+0000 2024-09-15T20:39:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:25.851 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:25.851 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 1 Processor -- start 2024-09-15T20:39:25.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 1 -- start start 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient: init 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient(hunting): picked mon.noname-a con 0x7f9d78135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient(hunting): picked mon.noname-c con 0x7f9d780a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient(hunting): picked mon.noname-b con 0x7f9d7813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d780745e0 con 0x7f9d78135b60 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d7806ecf0 con 0x7f9d780a24e0 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d7806efd0 con 0x7f9d7813c220 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient(hunting): authenticate will time out at 1626.722834s 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7e2c2640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9d78135b60 0x7f9d78135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49410/0 (socket says 172.21.15.78:49410) 2024-09-15T20:39:25.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7e2c2640 1 -- 172.21.15.78:0/1078728109 learned_addr learned my addr 172.21.15.78:0/1078728109 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2563124694 0 0) 0x7f9d780745e0 con 0x7f9d78135b60 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient(hunting): my global_id is 104115 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d60003530 con 0x7f9d78135b60 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 126064064 0 0) 0x7f9d7806efd0 con 0x7f9d7813c220 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient(hunting): my global_id is 84452 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d780745e0 con 0x7f9d7813c220 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2318384163 0 0) 0x7f9d60003530 con 0x7f9d78135b60 2024-09-15T20:39:25.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 >> v1:172.21.15.78:6790/0 conn(0x7f9d7813c220 legacy=0x7f9d7813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 >> v1:172.21.15.146:6789/0 conn(0x7f9d780a24e0 legacy=0x7f9d7813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: _finish_hunting 0 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 monclient: found mon.noname-a 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d78136650 con 0x7f9d78135b60 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: _finish_auth 0 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: _check_auth_tickets 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d68001a20 con 0x7f9d78135b60 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:39:25.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: dump: 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 5 monclient: authenticate success, global_id 104115 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d68001bc0 con 0x7f9d78135b60 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:25.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 1 -- 172.21.15.78:0/1078728109 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d68001fd0 con 0x7f9d78135b60 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d7d2c0640 10 monclient: dump: 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.864 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.834+0000 7f9d80526640 4 set_mon_vals no callback set 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: shutdown 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- 172.21.15.78:0/1078728109 >> v1:172.21.15.78:6789/0 conn(0x7f9d78135b60 legacy=0x7f9d78135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- 172.21.15.78:0/1078728109 shutdown_connections 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- 172.21.15.78:0/1078728109 >> 172.21.15.78:0/1078728109 conn(0x7f9d780a63c0 msgr2=0x7f9d780a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- 172.21.15.78:0/1078728109 shutdown_connections 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- 172.21.15.78:0/1078728109 wait complete. 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: build_initial_monmap 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: monmap: 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:25.865 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 librados: starting msgr at 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 librados: starting objecter 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 Processor -- start 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- start start 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 librados: setting wanted keys 2024-09-15T20:39:25.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 librados: calling monclient init 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: init 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient(hunting): picked mon.noname-c con 0x7f9d780a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient(hunting): picked mon.noname-b con 0x7f9d78135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient(hunting): picked mon.noname-a con 0x7f9d7813c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d780729a0 con 0x7f9d7813c220 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d7809e490 con 0x7f9d78135b60 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9d7813bb70 con 0x7f9d780a24e0 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d7e2c2640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f9d780a24e0 0x7f9d78071b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51116/0 (socket says 172.21.15.78:51116) 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d7e2c2640 1 -- 172.21.15.78:0/3974075867 learned_addr learned my addr 172.21.15.78:0/3974075867 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient(hunting): authenticate will time out at 1626.726062s 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 client.?.objecter ms_handle_connect 0x7f9d780a24e0 2024-09-15T20:39:25.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 client.?.objecter ms_handle_connect 0x7f9d7813c220 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 772690221 0 0) 0x7f9d7813bb70 con 0x7f9d780a24e0 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): my global_id is 84455 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d64004ec0 con 0x7f9d780a24e0 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679930587 0 0) 0x7f9d780729a0 con 0x7f9d7813c220 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): my global_id is 104118 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d7813bb70 con 0x7f9d7813c220 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 client.?.objecter ms_handle_connect 0x7f9d78135b60 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3611542544 0 0) 0x7f9d64004ec0 con 0x7f9d780a24e0 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d780729a0 con 0x7f9d780a24e0 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d68001f50 con 0x7f9d780a24e0 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): dump: 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.869 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 378731962 0 0) 0x7f9d7813bb70 con 0x7f9d7813c220 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d64004ec0 con 0x7f9d7813c220 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d700012e0 con 0x7f9d7813c220 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:25.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): dump: 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1730166563 0 0) 0x7f9d7809e490 con 0x7f9d78135b60 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient(hunting): my global_id is 23512 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d7813bb70 con 0x7f9d78135b60 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2039619231 0 0) 0x7f9d780729a0 con 0x7f9d780a24e0 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 >> v1:172.21.15.146:6789/0 conn(0x7f9d78135b60 legacy=0x7f9d78072290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 >> v1:172.21.15.78:6789/0 conn(0x7f9d7813c220 legacy=0x7f9d781d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 monclient: found mon.c 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7813fa70 con 0x7f9d780a24e0 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient: _finish_auth 0 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:39:25.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 5 monclient: authenticate success, global_id 84455 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: _renew_subs 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7809e1b0 con 0x7f9d780a24e0 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 client.84455.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: _renew_subs 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9d7809ded0 con 0x7f9d780a24e0 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 librados: init done 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9d68001ba0 con 0x7f9d780a24e0 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d68005120 con 0x7f9d780a24e0 2024-09-15T20:39:25.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d7cabf640 4 set_mon_vals no callback set 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 10 monclient: dump: 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.874 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f9d68011190 con 0x7f9d780a24e0 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.838+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9d580065d0 con 0x7f9d58002870 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 client.84455.objecter ms_handle_connect 0x7f9d58002870 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 client.84455.objecter resend_mon_ops 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(339..339 src has 1..339) ==== 5346+0+0 (unknown 691184420 0 0) 0x7f9d6804bbb0 con 0x7f9d780a24e0 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 client.84455.objecter ms_dispatch 0x7f9d78000ba0 osd_map(339..339 src has 1..339) 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 3 client.84455.objecter handle_osd_map got epochs [339,339] > 0 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 3 client.84455.objecter handle_osd_map decoding full epoch 339 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 20 client.84455.objecter dump_active .. 0 homeless 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2990028458 0 0) 0x7f9d580065d0 con 0x7f9d58002870 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient(hunting): my global_id is 84458 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d780729a0 con 0x7f9d58002870 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2909003447 0 0) 0x7f9d780729a0 con 0x7f9d58002870 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d580065d0 con 0x7f9d58002870 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d6c001820 con 0x7f9d58002870 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d7eac3640 10 monclient: get_auth_request con 0x7f9d6403e290 auth_method 0 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient: dump: 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.876 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2247336294 0 0) 0x7f9d580065d0 con 0x7f9d58002870 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9d5800a170 con 0x7f9d58002870 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 client.84455.objecter ms_handle_connect 0x7f9d6403e290 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9d6c001ac0 con 0x7f9d58002870 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 >> v1:172.21.15.78:6790/0 conn(0x7f9d58002870 legacy=0x7f9d58004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.842+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 >> v1:172.21.15.78:6790/0 conn(0x7f9d58002870 legacy=0x7f9d58004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d80526640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9d58009e80 con 0x7f9d58009140 2024-09-15T20:39:25.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 client.84455.objecter ms_handle_connect 0x7f9d58009140 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 client.84455.objecter resend_mon_ops 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412308996 0 0) 0x7f9d58009e80 con 0x7f9d58009140 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient(hunting): my global_id is 84461 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9d580065d0 con 0x7f9d58009140 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2992836158 0 0) 0x7f9d580065d0 con 0x7f9d58009140 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9d58009e80 con 0x7f9d58009140 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9d6c0037a0 con 0x7f9d58009140 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:25.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient: dump: 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2143063018 0 0) 0x7f9d58009e80 con 0x7f9d58009140 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9d5800c420 con 0x7f9d58009140 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 2725311272) 0x7f9d6c005a80 con 0x7f9d58009140 2024-09-15T20:39:25.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 >> v1:172.21.15.78:6790/0 conn(0x7f9d58009140 legacy=0x7f9d580095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.846+0000 7f9d76ffd640 1 -- 172.21.15.78:0/3974075867 >> v1:172.21.15.78:6790/0 conn(0x7f9d58009140 legacy=0x7f9d580095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 32, 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 237533, 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:39:25.880 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:25.881 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:25.882 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:39:25.883 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:25.884 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.885 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:25.886 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.887 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.888 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.889 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:25.890 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:39:25.891 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 10 librados: watch_flush enter 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 10 librados: watch_flush exit 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 20 client.84455.objecter shutdown clearing up homeless session... 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 10 client.84455.objecter successfully canceled tick 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 >> v1:172.21.15.146:6812/435598642 conn(0x7f9d6403e290 legacy=0x7f9d64040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 10 monclient: shutdown 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 >> v1:172.21.15.78:6790/0 conn(0x7f9d780a24e0 legacy=0x7f9d78071b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d7eac3640 1 -- 172.21.15.78:0/3974075867 reap_dead start 2024-09-15T20:39:25.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d7eac3640 1 -- 172.21.15.78:0/3974075867 reap_dead start 2024-09-15T20:39:25.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 shutdown_connections 2024-09-15T20:39:25.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 >> 172.21.15.78:0/3974075867 conn(0x7f9d780a63c0 msgr2=0x7f9d7812e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:25.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 shutdown_connections 2024-09-15T20:39:25.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 1 -- 172.21.15.78:0/3974075867 wait complete. 2024-09-15T20:39:25.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:25.850+0000 7f9d80526640 1 librados: shutdown 2024-09-15T20:39:25.894 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:39:25.894 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:39:25.894 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:39:26.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.006+0000 7fec17ed0640 10 monclient: get_monmap_and_config 2024-09-15T20:39:26.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.006+0000 7fec17ed0640 10 monclient: build_initial_monmap 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.006+0000 7fec17ed0640 10 monclient: monmap: 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:26.011782+0000 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:26.011782+0000 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:26.011 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.006+0000 7fec17ed0640 1 Processor -- start 2024-09-15T20:39:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.006+0000 7fec17ed0640 1 -- start start 2024-09-15T20:39:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.006+0000 7fec17ed0640 10 monclient: init 2024-09-15T20:39:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:26.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): picked mon.noname-c con 0x7fec10134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): picked mon.noname-b con 0x7fec1009f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): picked mon.noname-a con 0x7fec1006e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec100a0470 con 0x7fec1006e960 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec1009feb0 con 0x7fec10134bb0 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec1009e4a0 con 0x7fec1009f5a0 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): authenticate will time out at 1626.896318s 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec1546b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fec1009f5a0 0x7fec101399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51146/0 (socket says 172.21.15.78:51146) 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec1646d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fec1006e960 0x7fec1013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49416/0 (socket says 172.21.15.78:49416) 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec1546b640 1 -- 172.21.15.78:0/3845987710 learned_addr learned my addr 172.21.15.78:0/3845987710 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104256693 0 0) 0x7fec0c001050 con 0x7fec1006e960 2024-09-15T20:39:26.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient(hunting): my global_id is 104121 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febfc003530 con 0x7fec1006e960 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1660253160 0 0) 0x7fec1009e4a0 con 0x7fec1009f5a0 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient(hunting): my global_id is 84464 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec0c001050 con 0x7fec1009f5a0 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2499866790 0 0) 0x7febfc003530 con 0x7fec1006e960 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 >> v1:172.21.15.78:6790/0 conn(0x7fec1009f5a0 legacy=0x7fec101399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 >> v1:172.21.15.146:6789/0 conn(0x7fec10134bb0 legacy=0x7fec10134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: _finish_hunting 0 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 monclient: found mon.noname-a 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec100a0ff0 con 0x7fec1006e960 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: _finish_auth 0 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: _check_auth_tickets 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 5 monclient: authenticate success, global_id 104121 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fec0c0017a0 con 0x7fec1006e960 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: dump: 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.022 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec0c001a30 con 0x7fec1006e960 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 1 -- 172.21.15.78:0/3845987710 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fec0c0047f0 con 0x7fec1006e960 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 4 set_mon_vals no callback set 2024-09-15T20:39:26.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec14c6a640 10 monclient: dump: 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: shutdown 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:26.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- 172.21.15.78:0/3845987710 >> v1:172.21.15.78:6789/0 conn(0x7fec1006e960 legacy=0x7fec1013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- 172.21.15.78:0/3845987710 shutdown_connections 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- 172.21.15.78:0/3845987710 >> 172.21.15.78:0/3845987710 conn(0x7fec100a63c0 msgr2=0x7fec100a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- 172.21.15.78:0/3845987710 shutdown_connections 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- 172.21.15.78:0/3845987710 wait complete. 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: build_initial_monmap 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: monmap: 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.025 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 librados: starting msgr at 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 librados: starting objecter 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 Processor -- start 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- start start 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 librados: setting wanted keys 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 librados: calling monclient init 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: init 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): picked mon.noname-b con 0x7fec1006e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:26.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): picked mon.noname-a con 0x7fec1009f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): picked mon.noname-c con 0x7fec10134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec1013c680 con 0x7fec1009f5a0 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec1009d180 con 0x7fec1006e960 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec100a4390 con 0x7fec10134bb0 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec17ed0640 10 monclient(hunting): authenticate will time out at 1626.899546s 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec1546b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fec1009f5a0 0x7fec101d1d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49426/0 (socket says 172.21.15.78:49426) 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec1546b640 1 -- 172.21.15.78:0/3758669184 learned_addr learned my addr 172.21.15.78:0/3758669184 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 client.?.objecter ms_handle_connect 0x7fec1009f5a0 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 client.?.objecter ms_handle_connect 0x7fec10134bb0 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4145494418 0 0) 0x7fec1013c680 con 0x7fec1009f5a0 2024-09-15T20:39:26.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 monclient(hunting): my global_id is 104124 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febf4004ef0 con 0x7fec1009f5a0 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037820684 0 0) 0x7fec100a4390 con 0x7fec10134bb0 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 10 monclient(hunting): my global_id is 84467 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.010+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec1013c680 con 0x7fec10134bb0 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 client.?.objecter ms_handle_connect 0x7fec1006e960 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2640844893 0 0) 0x7fec1013c680 con 0x7fec10134bb0 2024-09-15T20:39:26.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec100a4390 con 0x7fec10134bb0 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1449885375 0 0) 0x7febf4004ef0 con 0x7fec1009f5a0 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fec1013c680 con 0x7fec1009f5a0 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fec0c001be0 con 0x7fec10134bb0 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): dump: 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.029 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1680044498 0 0) 0x7fec1009d180 con 0x7fec1006e960 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): my global_id is 23518 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febf4004ef0 con 0x7fec1006e960 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fec00001900 con 0x7fec1009f5a0 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): dump: 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.030 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1433547823 0 0) 0x7fec100a4390 con 0x7fec10134bb0 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 >> v1:172.21.15.146:6789/0 conn(0x7fec1006e960 legacy=0x7fec101ccda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 >> v1:172.21.15.78:6789/0 conn(0x7fec1009f5a0 legacy=0x7fec101d1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: _finish_hunting 0 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 monclient: found mon.c 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:26.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec1013cc40 con 0x7fec10134bb0 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: _finish_auth 0 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: _check_auth_tickets 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 5 monclient: authenticate success, global_id 84467 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 10 monclient: _renew_subs 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fec1013b710 con 0x7fec10134bb0 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 10 client.84467.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 10 monclient: _renew_subs 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fec1006fe40 con 0x7fec10134bb0 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 1 librados: init done 2024-09-15T20:39:26.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fec0c001a30 con 0x7fec10134bb0 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fec0c004f00 con 0x7fec10134bb0 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec07fff640 4 set_mon_vals no callback set 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: dump: 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.033 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7febf80065d0 con 0x7febf8002870 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fec0c010f70 con 0x7fec10134bb0 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(339..339 src has 1..339) ==== 5346+0+0 (unknown 691184420 0 0) 0x7fec0c04b7b0 con 0x7fec10134bb0 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 client.84467.objecter ms_dispatch 0x7fec10000ba0 osd_map(339..339 src has 1..339) 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 3 client.84467.objecter handle_osd_map got epochs [339,339] > 0 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 3 client.84467.objecter handle_osd_map decoding full epoch 339 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 20 client.84467.objecter dump_active .. 0 homeless 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 client.84467.objecter ms_handle_connect 0x7febf8002870 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 client.84467.objecter resend_mon_ops 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec1546b640 10 monclient: get_auth_request con 0x7febf403e2c0 auth_method 0 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2638247113 0 0) 0x7febf80065d0 con 0x7febf8002870 2024-09-15T20:39:26.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient(hunting): my global_id is 23521 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fec100a4390 con 0x7febf8002870 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 client.84467.objecter ms_handle_connect 0x7febf403e2c0 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 473151014 0 0) 0x7fec100a4390 con 0x7febf8002870 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febf80065d0 con 0x7febf8002870 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fec08001ad0 con 0x7febf8002870 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: dump: 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.035 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2722094432 0 0) 0x7febf80065d0 con 0x7febf8002870 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7febf800a170 con 0x7febf8002870 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fec08001d10 con 0x7febf8002870 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:26.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 >> v1:172.21.15.146:6789/0 conn(0x7febf8002870 legacy=0x7febf8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.014+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 >> v1:172.21.15.146:6789/0 conn(0x7febf8002870 legacy=0x7febf8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:26.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec17ed0640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:39:26.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7febf8009e80 con 0x7febf8009140 2024-09-15T20:39:26.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 client.84467.objecter ms_handle_connect 0x7febf8009140 2024-09-15T20:39:26.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 client.84467.objecter resend_mon_ops 2024-09-15T20:39:26.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3368535080 0 0) 0x7febf8009e80 con 0x7febf8009140 2024-09-15T20:39:26.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 monclient(hunting): my global_id is 23524 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7febf80065d0 con 0x7febf8009140 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2113770774 0 0) 0x7febf80065d0 con 0x7febf8009140 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7febf8009e80 con 0x7febf8009140 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fec08003b20 con 0x7febf8009140 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 10 monclient: dump: 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.039 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2089795571 0 0) 0x7febf8009e80 con 0x7febf8009140 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.022+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7febf800c420 con 0x7febf8009140 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 276980199) 0x7fec08003ab0 con 0x7febf8009140 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec067fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec067fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 >> v1:172.21.15.146:6789/0 conn(0x7febf8009140 legacy=0x7febf80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec067fc640 1 -- 172.21.15.78:0/3758669184 >> v1:172.21.15.146:6789/0 conn(0x7febf8009140 legacy=0x7febf80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:39:26.040 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 32, 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 478534, 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:39:26.041 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:26.042 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:26.043 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:39:26.044 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:26.045 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.046 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:26.047 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:26.048 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.049 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:26.050 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:26.051 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:26.052 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:26.052 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:26.052 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:26.052 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:26.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:26.052 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:39:26.052 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 10 librados: watch_flush enter 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 10 librados: watch_flush exit 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 20 client.84467.objecter shutdown clearing up homeless session... 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 10 client.84467.objecter successfully canceled tick 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 >> v1:172.21.15.146:6812/435598642 conn(0x7febf403e2c0 legacy=0x7febf4040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 10 monclient: shutdown 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 >> v1:172.21.15.78:6790/0 conn(0x7fec10134bb0 legacy=0x7fec101d54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec1646d640 1 -- 172.21.15.78:0/3758669184 reap_dead start 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec1646d640 1 -- 172.21.15.78:0/3758669184 reap_dead start 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 shutdown_connections 2024-09-15T20:39:26.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 >> 172.21.15.78:0/3758669184 conn(0x7fec100a63c0 msgr2=0x7fec10073530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:26.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 shutdown_connections 2024-09-15T20:39:26.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 1 -- 172.21.15.78:0/3758669184 wait complete. 2024-09-15T20:39:26.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.026+0000 7fec17ed0640 1 librados: shutdown 2024-09-15T20:39:26.055 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:39:26.055 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:39:26.055 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:39:26.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.170+0000 7fe645028640 10 monclient: get_monmap_and_config 2024-09-15T20:39:26.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.170+0000 7fe645028640 10 monclient: build_initial_monmap 2024-09-15T20:39:26.175 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.170+0000 7fe645028640 10 monclient: monmap: 2024-09-15T20:39:26.175 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:26.175 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.175 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:26.176283+0000 2024-09-15T20:39:26.175 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:26.176283+0000 2024-09-15T20:39:26.175 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:26.176 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.176 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:26.176 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:26.176 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:26.176 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 Processor -- start 2024-09-15T20:39:26.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- start start 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: init 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): picked mon.noname-b con 0x7fe640134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): picked mon.noname-c con 0x7fe64009f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): picked mon.noname-a con 0x7fe64006e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6400a0470 con 0x7fe64006e960 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64009feb0 con 0x7fe64009f5a0 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64009e4a0 con 0x7fe640134bb0 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): authenticate will time out at 1627.060813s 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63ed9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe640134bb0 0x7fe640134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51166/0 (socket says 172.21.15.78:51166) 2024-09-15T20:39:26.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63ed9d640 1 -- 172.21.15.78:0/3363844891 learned_addr learned my addr 172.21.15.78:0/3363844891 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63f59e640 1 --1- 172.21.15.78:0/3363844891 >> v1:172.21.15.78:6789/0 conn(0x7fe64006e960 0x7fe64013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49428/0 (socket says 172.21.15.78:49428) 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1281248524 0 0) 0x7fe630001120 con 0x7fe64006e960 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient(hunting): my global_id is 104127 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe62c003530 con 0x7fe64006e960 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1557491012 0 0) 0x7fe64009e4a0 con 0x7fe640134bb0 2024-09-15T20:39:26.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient(hunting): my global_id is 84470 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe630001120 con 0x7fe640134bb0 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3165269503 0 0) 0x7fe62c003530 con 0x7fe64006e960 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 >> v1:172.21.15.78:6790/0 conn(0x7fe640134bb0 legacy=0x7fe640134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 >> v1:172.21.15.146:6789/0 conn(0x7fe64009f5a0 legacy=0x7fe6401399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: _finish_hunting 0 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 monclient: found mon.noname-a 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6400a0ff0 con 0x7fe64006e960 2024-09-15T20:39:26.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: _finish_auth 0 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: _check_auth_tickets 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe630001870 con 0x7fe64006e960 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: dump: 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.186 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 5 monclient: authenticate success, global_id 104127 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe630001a10 con 0x7fe64006e960 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 4 set_mon_vals no callback set 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 1 -- 172.21.15.78:0/3363844891 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe6300047f0 con 0x7fe64006e960 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63dd9b640 10 monclient: dump: 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.187 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: shutdown 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- 172.21.15.78:0/3363844891 >> v1:172.21.15.78:6789/0 conn(0x7fe64006e960 legacy=0x7fe64013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- 172.21.15.78:0/3363844891 shutdown_connections 2024-09-15T20:39:26.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- 172.21.15.78:0/3363844891 >> 172.21.15.78:0/3363844891 conn(0x7fe6400a63c0 msgr2=0x7fe6400a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- 172.21.15.78:0/3363844891 shutdown_connections 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- 172.21.15.78:0/3363844891 wait complete. 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: build_initial_monmap 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: monmap: 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:26.189 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 librados: starting msgr at 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 librados: starting objecter 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 Processor -- start 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- start start 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 librados: setting wanted keys 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 librados: calling monclient init 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: init 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): picked mon.noname-b con 0x7fe64006e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): picked mon.noname-a con 0x7fe64009f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): picked mon.noname-c con 0x7fe640134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:26.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64011cb40 con 0x7fe64009f5a0 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe64011ce20 con 0x7fe64006e960 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe640073120 con 0x7fe640134bb0 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe645028640 10 monclient(hunting): authenticate will time out at 1627.063697s 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63f59e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe640134bb0 0x7fe6401d3710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51172/0 (socket says 172.21.15.78:51172) 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe63f59e640 1 -- 172.21.15.78:0/1360063362 learned_addr learned my addr 172.21.15.78:0/1360063362 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe6377fe640 10 client.?.objecter ms_handle_connect 0x7fe64009f5a0 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe6377fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe6377fe640 10 client.?.objecter ms_handle_connect 0x7fe640134bb0 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.174+0000 7fe6377fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2775713299 0 0) 0x7fe64011cb40 con 0x7fe64009f5a0 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): my global_id is 104130 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe620004ec0 con 0x7fe64009f5a0 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1496364315 0 0) 0x7fe640073120 con 0x7fe640134bb0 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): my global_id is 84473 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64011cb40 con 0x7fe640134bb0 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 client.?.objecter ms_handle_connect 0x7fe64006e960 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2861956025 0 0) 0x7fe620004ec0 con 0x7fe64009f5a0 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe640073120 con 0x7fe64009f5a0 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe628001cc0 con 0x7fe64009f5a0 2024-09-15T20:39:26.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): dump: 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:26.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1279336228 0 0) 0x7fe64011cb40 con 0x7fe640134bb0 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe620004ec0 con 0x7fe640134bb0 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe630000ff0 con 0x7fe640134bb0 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): dump: 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.194 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 681172248 0 0) 0x7fe64011ce20 con 0x7fe64006e960 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient(hunting): my global_id is 23530 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe64011cb40 con 0x7fe64006e960 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1338428727 0 0) 0x7fe640073120 con 0x7fe64009f5a0 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 >> v1:172.21.15.78:6790/0 conn(0x7fe640134bb0 legacy=0x7fe6401d3710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 >> v1:172.21.15.146:6789/0 conn(0x7fe64006e960 legacy=0x7fe6401331b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient: _finish_hunting 0 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 monclient: found mon.a 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe64011d100 con 0x7fe64009f5a0 2024-09-15T20:39:26.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient: _finish_auth 0 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient: _check_auth_tickets 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 5 monclient: authenticate success, global_id 104130 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 10 monclient: _renew_subs 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe64011c860 con 0x7fe64009f5a0 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 10 client.104130.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 10 monclient: _renew_subs 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fe64013c500 con 0x7fe64009f5a0 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 1 librados: init done 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe628003980 con 0x7fe64009f5a0 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:26.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe628003cd0 con 0x7fe64009f5a0 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 monclient: dump: 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:26.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe63d59a640 4 set_mon_vals no callback set 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fe62800fd40 con 0x7fe64009f5a0 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe624005090 con 0x7fe64009f5a0 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(339..339 src has 1..339) ==== 5346+0+0 (unknown 691184420 0 0) 0x7fe62804aa20 con 0x7fe64009f5a0 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 client.104130.objecter ms_dispatch 0x7fe640000ba0 osd_map(339..339 src has 1..339) 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 3 client.104130.objecter handle_osd_map got epochs [339,339] > 0 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 3 client.104130.objecter handle_osd_map decoding full epoch 339 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 20 client.104130.objecter dump_active .. 0 homeless 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe63ed9d640 10 monclient: get_auth_request con 0x7fe62003e210 auth_method 0 2024-09-15T20:39:26.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.178+0000 7fe6377fe640 10 client.104130.objecter ms_handle_connect 0x7fe62003e210 2024-09-15T20:39:26.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.182+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fe628015ce0 con 0x7fe64009f5a0 2024-09-15T20:39:26.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.182+0000 7fe6377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:26.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.182+0000 7fe6377fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:26.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.410+0000 7fe645028640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:39:26.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.410+0000 7fe645028640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:39:26.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.410+0000 7fe645028640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:26.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.410+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe624005c50 con 0x7fe64009f5a0 2024-09-15T20:39:26.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe6377fe640 1 -- 172.21.15.78:0/1360063362 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7fe6280155e0 con 0x7fe64009f5a0 2024-09-15T20:39:26.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe6377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:39:26.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe6377fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:26.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1847943773,config=2586702516,health=1995790572,kv=199072906,logm=3709066873}) 2024-09-15T20:39:26.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=15,config=2,health=9,kv=12,logm=62} crc {auth=2840433967,config=2586702516,health=1470180416,kv=199072906,logm=4280480407}) 2024-09-15T20:39:26.418 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.418 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=16,config=2,health=12,kv=12,logm=58} crc {auth=1847943773,config=2586702516,health=1995790572,kv=199072906,logm=3709066873}) 2024-09-15T20:39:26.418 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=16,config=1,health=12,kv=12,logm=59} crc {auth=1322608081,config=1228700967,health=1995790572,kv=852237645,logm=3954560935}) 2024-09-15T20:39:26.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 10 librados: watch_flush enter 2024-09-15T20:39:26.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 10 librados: watch_flush exit 2024-09-15T20:39:26.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 20 client.104130.objecter shutdown clearing up homeless session... 2024-09-15T20:39:26.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 10 client.104130.objecter successfully canceled tick 2024-09-15T20:39:26.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 >> v1:172.21.15.146:6812/435598642 conn(0x7fe62003e210 legacy=0x7fe6200406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 10 monclient: shutdown 2024-09-15T20:39:26.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:26.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 >> v1:172.21.15.78:6789/0 conn(0x7fe64009f5a0 legacy=0x7fe6401cffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:26.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 shutdown_connections 2024-09-15T20:39:26.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 >> 172.21.15.78:0/1360063362 conn(0x7fe6400a63c0 msgr2=0x7fe640073610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:26.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 shutdown_connections 2024-09-15T20:39:26.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 1 -- 172.21.15.78:0/1360063362 wait complete. 2024-09-15T20:39:26.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:26.414+0000 7fe645028640 1 librados: shutdown 2024-09-15T20:39:26.420 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.420 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799997593}) 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2314022029}) 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799997593}) 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=13948855}) 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657387033}) 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1355475229}) 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=657387033}) 2024-09-15T20:39:26.422 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.418+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1642873197}) 2024-09-15T20:39:26.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.422+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.422+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2782620453}) 2024-09-15T20:39:26.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.422+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1535618106}) 2024-09-15T20:39:26.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.422+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.422+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2782620453}) 2024-09-15T20:39:26.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.422+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3101412901}) 2024-09-15T20:39:26.428 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.426+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.428 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.426+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799244025}) 2024-09-15T20:39:26.428 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.426+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2644450442}) 2024-09-15T20:39:26.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.426+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.426+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3799244025}) 2024-09-15T20:39:26.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.426+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771677918}) 2024-09-15T20:39:26.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.432 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119769520}) 2024-09-15T20:39:26.432 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1104135865}) 2024-09-15T20:39:26.432 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.432 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1119769520}) 2024-09-15T20:39:26.432 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2898215399}) 2024-09-15T20:39:26.434 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.434 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133841174}) 2024-09-15T20:39:26.434 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1209348724}) 2024-09-15T20:39:26.434 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.434 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=133841174}) 2024-09-15T20:39:26.434 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.430+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3681188277}) 2024-09-15T20:39:26.436 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.436 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024795429}) 2024-09-15T20:39:26.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258462434}) 2024-09-15T20:39:26.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1024795429}) 2024-09-15T20:39:26.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3156794591}) 2024-09-15T20:39:26.438 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:39:26.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518323483}) 2024-09-15T20:39:26.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1389746458}) 2024-09-15T20:39:26.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2518323483}) 2024-09-15T20:39:26.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.434+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1967257984}) 2024-09-15T20:39:26.441 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.441 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018166991}) 2024-09-15T20:39:26.441 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2627345610}) 2024-09-15T20:39:26.441 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3018166991}) 2024-09-15T20:39:26.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3038902622}) 2024-09-15T20:39:26.443 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.443 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=904110985}) 2024-09-15T20:39:26.443 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2406541219}) 2024-09-15T20:39:26.443 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.443 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=904110985}) 2024-09-15T20:39:26.443 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.438+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=162858984}) 2024-09-15T20:39:26.445 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.445 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716488540}) 2024-09-15T20:39:26.445 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1075753947}) 2024-09-15T20:39:26.445 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.445 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3716488540}) 2024-09-15T20:39:26.445 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=541013964}) 2024-09-15T20:39:26.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455142618}) 2024-09-15T20:39:26.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=775862156}) 2024-09-15T20:39:26.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3455142618}) 2024-09-15T20:39:26.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.442+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=789005248}) 2024-09-15T20:39:26.448 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.449 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581217355}) 2024-09-15T20:39:26.449 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1467069683}) 2024-09-15T20:39:26.449 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.449 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=581217355}) 2024-09-15T20:39:26.449 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2748667404}) 2024-09-15T20:39:26.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228398981}) 2024-09-15T20:39:26.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1198463240}) 2024-09-15T20:39:26.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=228398981}) 2024-09-15T20:39:26.451 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.446+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4036133601}) 2024-09-15T20:39:26.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035934199}) 2024-09-15T20:39:26.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1556776382}) 2024-09-15T20:39:26.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4035934199}) 2024-09-15T20:39:26.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4292723143}) 2024-09-15T20:39:26.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946674267}) 2024-09-15T20:39:26.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4177920938}) 2024-09-15T20:39:26.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946674267}) 2024-09-15T20:39:26.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2262859159}) 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230354024}) 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2384538776}) 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4230354024}) 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.450+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1985985933}) 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085055356}) 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3637153761}) 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085055356}) 2024-09-15T20:39:26.458 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1230735660}) 2024-09-15T20:39:26.459 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.459 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096712925}) 2024-09-15T20:39:26.459 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3319734420}) 2024-09-15T20:39:26.459 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.459 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1096712925}) 2024-09-15T20:39:26.460 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.454+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2067321335}) 2024-09-15T20:39:26.460 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.461 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605407920}) 2024-09-15T20:39:26.461 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2292265252}) 2024-09-15T20:39:26.461 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.461 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=605407920}) 2024-09-15T20:39:26.461 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1043975227}) 2024-09-15T20:39:26.462 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.462 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1571912988}) 2024-09-15T20:39:26.462 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3484137470}) 2024-09-15T20:39:26.463 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.463 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1571912988}) 2024-09-15T20:39:26.463 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.458+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4136699353}) 2024-09-15T20:39:26.464 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.464 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334230800}) 2024-09-15T20:39:26.464 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1606489767}) 2024-09-15T20:39:26.464 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.464 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1334230800}) 2024-09-15T20:39:26.464 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1624159886}) 2024-09-15T20:39:26.465 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.465 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946219416}) 2024-09-15T20:39:26.466 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4156878310}) 2024-09-15T20:39:26.466 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.466 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=946219416}) 2024-09-15T20:39:26.466 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2459081490}) 2024-09-15T20:39:26.467 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.467 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727420671}) 2024-09-15T20:39:26.467 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3880240472}) 2024-09-15T20:39:26.467 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.467 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2727420671}) 2024-09-15T20:39:26.467 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.462+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4279751096}) 2024-09-15T20:39:26.469 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.469 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833495357}) 2024-09-15T20:39:26.469 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2106425926}) 2024-09-15T20:39:26.469 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.469 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3833495357}) 2024-09-15T20:39:26.469 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2597366502}) 2024-09-15T20:39:26.470 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.470 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918118332}) 2024-09-15T20:39:26.470 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3922917892}) 2024-09-15T20:39:26.470 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.470 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2918118332}) 2024-09-15T20:39:26.470 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.466+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702657890}) 2024-09-15T20:39:26.472 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.472 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279853116}) 2024-09-15T20:39:26.472 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3775147506}) 2024-09-15T20:39:26.472 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.472 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1279853116}) 2024-09-15T20:39:26.472 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3622177002}) 2024-09-15T20:39:26.473 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.473 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345652571}) 2024-09-15T20:39:26.473 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=978767424}) 2024-09-15T20:39:26.474 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.474 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2345652571}) 2024-09-15T20:39:26.474 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1578500325}) 2024-09-15T20:39:26.475 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.475 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928593467}) 2024-09-15T20:39:26.475 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2434660979}) 2024-09-15T20:39:26.475 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.475 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=928593467}) 2024-09-15T20:39:26.475 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.470+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2711939809}) 2024-09-15T20:39:26.477 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.477 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122947433}) 2024-09-15T20:39:26.477 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3203646114}) 2024-09-15T20:39:26.477 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.477 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2122947433}) 2024-09-15T20:39:26.477 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671140045}) 2024-09-15T20:39:26.478 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.478 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848221340}) 2024-09-15T20:39:26.478 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2982363596}) 2024-09-15T20:39:26.478 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.478 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3848221340}) 2024-09-15T20:39:26.478 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2955710448}) 2024-09-15T20:39:26.479 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.479 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459277700}) 2024-09-15T20:39:26.480 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.474+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3590485902}) 2024-09-15T20:39:26.480 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.480 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1459277700}) 2024-09-15T20:39:26.480 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3447865078}) 2024-09-15T20:39:26.481 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.481 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=502205847}) 2024-09-15T20:39:26.481 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3530867258}) 2024-09-15T20:39:26.481 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.481 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=502205847}) 2024-09-15T20:39:26.481 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4033566187}) 2024-09-15T20:39:26.482 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.482 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202071869}) 2024-09-15T20:39:26.482 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1261322880}) 2024-09-15T20:39:26.482 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.482 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2202071869}) 2024-09-15T20:39:26.483 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.478+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3048261560}) 2024-09-15T20:39:26.484 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.484 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426970128}) 2024-09-15T20:39:26.484 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3147313719}) 2024-09-15T20:39:26.484 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.484 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3426970128}) 2024-09-15T20:39:26.484 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4077448305}) 2024-09-15T20:39:26.486 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.486 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=9,mon_config_key=5} crc {logm=2080640603,mdsmap=503983409,mgr=2617012046,mgr_metadata=1997488951,mgrstat=658432104,mon_config_key=1054573076}) 2024-09-15T20:39:26.486 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=71,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=6} crc {logm=549149117,mdsmap=503983409,mgr=4145042074,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=629676316,mon_config_key=3957053206}) 2024-09-15T20:39:26.486 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.486 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=74,mdsmap=3,mgr=8,mgr_metadata=1,mgrstat=9,mon_config_key=5} crc {logm=2080640603,mdsmap=503983409,mgr=2617012046,mgr_metadata=1997488951,mgrstat=658432104,mon_config_key=1054573076}) 2024-09-15T20:39:26.486 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.482+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=75,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=1} crc {logm=156664517,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=629676316,mon_config_key=2197556201}) 2024-09-15T20:39:26.488 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.488 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=49} crc {mon_config_key=4065930088,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=1930733703,osd_snap=49882393,osdmap=2958001734}) 2024-09-15T20:39:26.488 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=29,osdmap=47} crc {mon_config_key=4065930088,monmap=3785329752,nvmeofgw=598337686,osd_metadata=159488793,osd_pg_creating=1930733703,osd_snap=2191528956,osdmap=3298110728}) 2024-09-15T20:39:26.488 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.488 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=12,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=26,osdmap=49} crc {mon_config_key=4065930088,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=1930733703,osd_snap=49882393,osdmap=2958001734}) 2024-09-15T20:39:26.488 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=27,osdmap=50} crc {mon_config_key=2321550713,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=1930733703,osd_snap=1741372224,osdmap=3427634263}) 2024-09-15T20:39:26.490 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.490 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3103825023}) 2024-09-15T20:39:26.490 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3672948790}) 2024-09-15T20:39:26.490 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.490 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3103825023}) 2024-09-15T20:39:26.490 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3706585795}) 2024-09-15T20:39:26.491 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.491 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3850155127}) 2024-09-15T20:39:26.491 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1437729430}) 2024-09-15T20:39:26.491 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.492 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3850155127}) 2024-09-15T20:39:26.492 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.486+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3147143160}) 2024-09-15T20:39:26.494 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.490+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.494 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.490+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1617610662}) 2024-09-15T20:39:26.494 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.490+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1980246955}) 2024-09-15T20:39:26.494 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.490+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.494 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.490+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1617610662}) 2024-09-15T20:39:26.494 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.490+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=241991292}) 2024-09-15T20:39:26.497 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.494+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.497 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.494+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2794876313}) 2024-09-15T20:39:26.497 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.494+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=154458183}) 2024-09-15T20:39:26.497 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.494+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.498 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.494+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2794876313}) 2024-09-15T20:39:26.498 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.494+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1300010606}) 2024-09-15T20:39:26.500 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.498+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.500 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.498+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=2420127535}) 2024-09-15T20:39:26.500 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.498+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=51} crc {osdmap=1033966838}) 2024-09-15T20:39:26.500 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.498+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:39:26.500 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.498+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=50} crc {osdmap=2420127535}) 2024-09-15T20:39:26.500 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:39:26.498+0000 7fe9232ae640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=49} crc {osdmap=688046686}) 2024-09-15T20:39:27.438 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:39:27.438 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:39:27.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient: get_monmap_and_config 2024-09-15T20:39:27.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient: build_initial_monmap 2024-09-15T20:39:27.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient: monmap: 2024-09-15T20:39:27.569 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:27.569 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.570 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:27.570516+0000 2024-09-15T20:39:27.570 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:27.570516+0000 2024-09-15T20:39:27.570 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:27.570 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.570 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:27.570 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:27.570 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:27.570 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 1 Processor -- start 2024-09-15T20:39:27.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 1 -- start start 2024-09-15T20:39:27.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient: init 2024-09-15T20:39:27.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:27.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:39:27.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient(hunting): picked mon.noname-c con 0x7f3f28137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:27.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient(hunting): picked mon.noname-b con 0x7f3f28074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient(hunting): picked mon.noname-a con 0x7f3f2813e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f28138850 con 0x7f3f2813e370 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f280a5da0 con 0x7f3f28137d60 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f28073560 con 0x7f3f28074500 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f2d452640 10 monclient(hunting): authenticate will time out at 1628.455369s 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f267fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f3f28074500 0x7f3f2813cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51184/0 (socket says 172.21.15.78:51184) 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f267fc640 1 -- 172.21.15.78:0/902846777 learned_addr learned my addr 172.21.15.78:0/902846777 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1194358406 0 0) 0x7f3f28073560 con 0x7f3f28074500 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 10 monclient(hunting): my global_id is 84476 2024-09-15T20:39:27.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f10003530 con 0x7f3f28074500 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3038637244 0 0) 0x7f3f28138850 con 0x7f3f2813e370 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 10 monclient(hunting): my global_id is 104133 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.566+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f28073560 con 0x7f3f2813e370 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3441483888 0 0) 0x7f3f10003530 con 0x7f3f28074500 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 >> v1:172.21.15.146:6789/0 conn(0x7f3f28137d60 legacy=0x7f3f28138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 >> v1:172.21.15.78:6789/0 conn(0x7f3f2813e370 legacy=0x7f3f28140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 1 monclient: found mon.noname-b 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:27.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f28073840 con 0x7f3f28074500 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: _finish_auth 0 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f180016a0 con 0x7f3f28074500 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: dump: 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.576 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 5 monclient: authenticate success, global_id 84476 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:27.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f180018c0 con 0x7f3f28074500 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 4 set_mon_vals no callback set 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 1 -- 172.21.15.78:0/902846777 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f18001cd0 con 0x7f3f28074500 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f25ffb640 10 monclient: dump: 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.578 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: shutdown 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- 172.21.15.78:0/902846777 >> v1:172.21.15.78:6790/0 conn(0x7f3f28074500 legacy=0x7f3f2813cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- 172.21.15.78:0/902846777 shutdown_connections 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- 172.21.15.78:0/902846777 >> 172.21.15.78:0/902846777 conn(0x7f3f280aa650 msgr2=0x7f3f280aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- 172.21.15.78:0/902846777 shutdown_connections 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- 172.21.15.78:0/902846777 wait complete. 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: build_initial_monmap 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: monmap: 2024-09-15T20:39:27.579 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 librados: starting msgr at 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 librados: starting objecter 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 Processor -- start 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- start start 2024-09-15T20:39:27.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 librados: setting wanted keys 2024-09-15T20:39:27.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 librados: calling monclient init 2024-09-15T20:39:27.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: init 2024-09-15T20:39:27.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient(hunting): picked mon.noname-a con 0x7f3f28074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient(hunting): picked mon.noname-c con 0x7f3f28137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient(hunting): picked mon.noname-b con 0x7f3f2813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f281417f0 con 0x7f3f28074500 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f28132620 con 0x7f3f2813e370 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3f28132900 con 0x7f3f28137d60 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient(hunting): authenticate will time out at 1628.458561s 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f267fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f3f28137d60 0x7f3f2809ef70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51188/0 (socket says 172.21.15.78:51188) 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f267fc640 1 -- 172.21.15.78:0/2158141881 learned_addr learned my addr 172.21.15.78:0/2158141881 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 client.?.objecter ms_handle_connect 0x7f3f28137d60 2024-09-15T20:39:27.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 client.?.objecter ms_handle_connect 0x7f3f28074500 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035331170 0 0) 0x7f3f281417f0 con 0x7f3f28074500 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): my global_id is 104136 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f14004ec0 con 0x7f3f28074500 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1407806093 0 0) 0x7f3f28132900 con 0x7f3f28137d60 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): my global_id is 84479 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f281417f0 con 0x7f3f28137d60 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 client.?.objecter ms_handle_connect 0x7f3f2813e370 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2494277590 0 0) 0x7f3f14004ec0 con 0x7f3f28074500 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f28132900 con 0x7f3f28074500 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f1c001940 con 0x7f3f28074500 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): dump: 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.587 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2705079403 0 0) 0x7f3f281417f0 con 0x7f3f28137d60 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3f14004ec0 con 0x7f3f28137d60 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f18001c10 con 0x7f3f28137d60 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): dump: 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.588 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3468004985 0 0) 0x7f3f28132620 con 0x7f3f2813e370 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient(hunting): my global_id is 23536 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3f281417f0 con 0x7f3f2813e370 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 502850939 0 0) 0x7f3f28132900 con 0x7f3f28074500 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 >> v1:172.21.15.78:6790/0 conn(0x7f3f28137d60 legacy=0x7f3f2809ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 >> v1:172.21.15.146:6789/0 conn(0x7f3f2813e370 legacy=0x7f3f2809f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 monclient: found mon.a 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f280a0180 con 0x7f3f28074500 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 5 monclient: authenticate success, global_id 104136 2024-09-15T20:39:27.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: _renew_subs 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3f280a0c60 con 0x7f3f28074500 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 client.104136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: _renew_subs 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3f281d5fe0 con 0x7f3f28074500 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f2d452640 1 librados: init done 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3f1c003600 con 0x7f3f28074500 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3f1c003950 con 0x7f3f28074500 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:27.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f257fa640 4 set_mon_vals no callback set 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.570+0000 7f3f0f7fe640 10 monclient: dump: 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f2d452640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:27.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f2d452640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f2d452640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f08005090 con 0x7f3f28074500 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f3f1c00f9c0 con 0x7f3f28074500 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(341..341 src has 1..341) ==== 5333+0+0 (unknown 208272338 0 0) 0x7f3f1c04a540 con 0x7f3f28074500 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f0f7fe640 10 client.104136.objecter ms_dispatch 0x7f3f28000ba0 osd_map(341..341 src has 1..341) 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f0f7fe640 3 client.104136.objecter handle_osd_map got epochs [341,341] > 0 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f0f7fe640 3 client.104136.objecter handle_osd_map decoding full epoch 341 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f0f7fe640 20 client.104136.objecter dump_active .. 0 homeless 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f267fc640 10 monclient: get_auth_request con 0x7f3f1403e290 auth_method 0 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.574+0000 7f3f0f7fe640 10 client.104136.objecter ms_handle_connect 0x7f3f1403e290 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.578+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f3f1c0158b0 con 0x7f3f28074500 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.578+0000 7f3f0f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:27.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.578+0000 7f3f0f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f2d452640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:39:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f2d452640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f2d452640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3f08005380 con 0x7f3f28074500 2024-09-15T20:39:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f0f7fe640 1 -- 172.21.15.78:0/2158141881 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 1416062709) 0x7f3f1c0151b0 con 0x7f3f28074500 2024-09-15T20:39:27.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f0f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:27.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f0f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:27.818 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:39:27.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f2d452640 10 librados: watch_flush enter 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f2d452640 10 librados: watch_flush exit 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f2d452640 20 client.104136.objecter shutdown clearing up homeless session... 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.814+0000 7f3f2d452640 10 client.104136.objecter successfully canceled tick 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 >> v1:172.21.15.146:6812/435598642 conn(0x7f3f1403e290 legacy=0x7f3f14040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 10 monclient: shutdown 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 >> v1:172.21.15.78:6789/0 conn(0x7f3f28074500 legacy=0x7f3f2809e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 shutdown_connections 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 >> 172.21.15.78:0/2158141881 conn(0x7f3f280aa650 msgr2=0x7f3f28130b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 shutdown_connections 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 1 -- 172.21.15.78:0/2158141881 wait complete. 2024-09-15T20:39:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.818+0000 7f3f2d452640 1 librados: shutdown 2024-09-15T20:39:27.835 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:39:27.835 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:39:27.835 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:39:27.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:39:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient: get_monmap_and_config 2024-09-15T20:39:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient: build_initial_monmap 2024-09-15T20:39:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient: monmap: 2024-09-15T20:39:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:27.957489+0000 2024-09-15T20:39:27.956 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:27.957489+0000 2024-09-15T20:39:27.957 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:27.957 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.957 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:27.957 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:27.957 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:27.957 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 1 Processor -- start 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 1 -- start start 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient: init 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac4134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac409f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac406e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac40a0470 con 0x7f9ac4134bb0 2024-09-15T20:39:27.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac409feb0 con 0x7f9ac409f5a0 2024-09-15T20:39:27.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac409e4a0 con 0x7f9ac406e960 2024-09-15T20:39:27.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:27.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient(hunting): authenticate will time out at 1628.842607s 2024-09-15T20:39:27.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac359e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f9ac406e960 0x7f9ac413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51192/0 (socket says 172.21.15.78:51192) 2024-09-15T20:39:27.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac359e640 1 -- 172.21.15.78:0/3551246846 learned_addr learned my addr 172.21.15.78:0/3551246846 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1260876636 0 0) 0x7f9ac409e4a0 con 0x7f9ac406e960 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient(hunting): my global_id is 84482 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab0003530 con 0x7f9ac406e960 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3295666032 0 0) 0x7f9ab4001120 con 0x7f9ac4134bb0 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient(hunting): my global_id is 104139 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac409e4a0 con 0x7f9ac4134bb0 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3044075805 0 0) 0x7f9ab0003530 con 0x7f9ac406e960 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 >> v1:172.21.15.146:6789/0 conn(0x7f9ac409f5a0 legacy=0x7f9ac41399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 >> v1:172.21.15.78:6789/0 conn(0x7f9ac4134bb0 legacy=0x7f9ac4134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: _finish_hunting 0 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 monclient: found mon.noname-b 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40a0ff0 con 0x7f9ac406e960 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: _finish_auth 0 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: _check_auth_tickets 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:27.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 5 monclient: authenticate success, global_id 84482 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9abc001a20 con 0x7f9ac406e960 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: dump: 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.968 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9abc001c40 con 0x7f9ac406e960 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 1 -- 172.21.15.78:0/3551246846 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9abc0020d0 con 0x7f9ac406e960 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac90e9640 4 set_mon_vals no callback set 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.954+0000 7f9ac259c640 10 monclient: dump: 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.969 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: shutdown 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- 172.21.15.78:0/3551246846 >> v1:172.21.15.78:6790/0 conn(0x7f9ac406e960 legacy=0x7f9ac413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- 172.21.15.78:0/3551246846 shutdown_connections 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- 172.21.15.78:0/3551246846 >> 172.21.15.78:0/3551246846 conn(0x7f9ac40a63c0 msgr2=0x7f9ac40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- 172.21.15.78:0/3551246846 shutdown_connections 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- 172.21.15.78:0/3551246846 wait complete. 2024-09-15T20:39:27.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: build_initial_monmap 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: monmap: 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 librados: starting msgr at 2024-09-15T20:39:27.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 librados: starting objecter 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 Processor -- start 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- start start 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 librados: setting wanted keys 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 librados: calling monclient init 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: init 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient(hunting): picked mon.noname-a con 0x7f9ac406e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient(hunting): picked mon.noname-c con 0x7f9ac409f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient(hunting): picked mon.noname-b con 0x7f9ac4134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac40a4870 con 0x7f9ac406e960 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac411e9c0 con 0x7f9ac4134bb0 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9ac411d400 con 0x7f9ac409f5a0 2024-09-15T20:39:27.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient(hunting): authenticate will time out at 1628.845738s 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac2d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9ac406e960 0x7f9ac411c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49462/0 (socket says 172.21.15.78:49462) 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac2d9d640 1 -- 172.21.15.78:0/1399428205 learned_addr learned my addr 172.21.15.78:0/1399428205 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abbfff640 1 --1- 172.21.15.78:0/1399428205 >> v1:172.21.15.78:6790/0 conn(0x7f9ac409f5a0 0x7f9ac411ccf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51196/0 (socket says 172.21.15.78:51196) 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 client.?.objecter ms_handle_connect 0x7f9ac409f5a0 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 client.?.objecter ms_handle_connect 0x7f9ac406e960 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1361155856 0 0) 0x7f9ac411d400 con 0x7f9ac409f5a0 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): my global_id is 84485 2024-09-15T20:39:27.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aa4004ef0 con 0x7f9ac409f5a0 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 client.?.objecter ms_handle_connect 0x7f9ac4134bb0 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 467339290 0 0) 0x7f9ab4001110 con 0x7f9ac406e960 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): my global_id is 104142 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ac411d400 con 0x7f9ac406e960 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 28428096 0 0) 0x7f9aa4004ef0 con 0x7f9ac409f5a0 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ab4001110 con 0x7f9ac409f5a0 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9aac001980 con 0x7f9ac409f5a0 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:27.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): dump: 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3398865494 0 0) 0x7f9ac411e9c0 con 0x7f9ac4134bb0 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): my global_id is 23542 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aa4004ef0 con 0x7f9ac4134bb0 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1268927598 0 0) 0x7f9ac411d400 con 0x7f9ac406e960 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9ac411e9c0 con 0x7f9ac406e960 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9ab4001570 con 0x7f9ac406e960 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): dump: 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.976 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1000494064 0 0) 0x7f9ab4001110 con 0x7f9ac409f5a0 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 >> v1:172.21.15.146:6789/0 conn(0x7f9ac4134bb0 legacy=0x7f9ac41d4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 >> v1:172.21.15.78:6789/0 conn(0x7f9ac406e960 legacy=0x7f9ac411c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 monclient: found mon.c 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ac40a42b0 con 0x7f9ac409f5a0 2024-09-15T20:39:27.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient: _finish_auth 0 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 5 monclient: authenticate success, global_id 84485 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: _renew_subs 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9ac409d180 con 0x7f9ac409f5a0 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 client.84485.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: _renew_subs 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f9ac413d0b0 con 0x7f9ac409f5a0 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 librados: init done 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9aac003620 con 0x7f9ac409f5a0 2024-09-15T20:39:27.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9aac0039f0 con 0x7f9ac409f5a0 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac1d9b640 4 set_mon_vals no callback set 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient: dump: 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.979 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9aa80065d0 con 0x7f9aa8002870 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 client.84485.objecter ms_handle_connect 0x7f9aa8002870 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 client.84485.objecter resend_mon_ops 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2631364628 0 0) 0x7f9aa80065d0 con 0x7f9aa8002870 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 10 monclient(hunting): my global_id is 104145 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9ab4001110 con 0x7f9aa8002870 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.958+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f9aac003c10 con 0x7f9ac409f5a0 2024-09-15T20:39:27.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(341..341 src has 1..341) ==== 5333+0+0 (unknown 208272338 0 0) 0x7f9aac04b710 con 0x7f9ac409f5a0 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 10 client.84485.objecter ms_dispatch 0x7f9ac4000ba0 osd_map(341..341 src has 1..341) 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 3 client.84485.objecter handle_osd_map got epochs [341,341] > 0 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 3 client.84485.objecter handle_osd_map decoding full epoch 341 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 20 client.84485.objecter dump_active .. 0 homeless 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3519397781 0 0) 0x7f9ab4001110 con 0x7f9aa8002870 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9aa80065d0 con 0x7f9aa8002870 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9ab40037a0 con 0x7f9aa8002870 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 10 monclient: dump: 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.981 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9ac359e640 10 monclient: get_auth_request con 0x7f9aa403e240 auth_method 0 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3585903937 0 0) 0x7f9aa80065d0 con 0x7f9aa8002870 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9aa800a170 con 0x7f9aa8002870 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 10 client.84485.objecter ms_handle_connect 0x7f9aa403e240 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9ab4005790 con 0x7f9aa8002870 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 >> v1:172.21.15.78:6789/0 conn(0x7f9aa8002870 legacy=0x7f9aa8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.962+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 >> v1:172.21.15.78:6789/0 conn(0x7f9aa8002870 legacy=0x7f9aa8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:27.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9ac90e9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:39:27.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9aa8009e80 con 0x7f9aa8009140 2024-09-15T20:39:27.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 client.84485.objecter ms_handle_connect 0x7f9aa8009140 2024-09-15T20:39:27.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 client.84485.objecter resend_mon_ops 2024-09-15T20:39:27.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 233436904 0 0) 0x7f9aa8009e80 con 0x7f9aa8009140 2024-09-15T20:39:27.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:27.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient(hunting): my global_id is 104148 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9aa80065d0 con 0x7f9aa8009140 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3648948905 0 0) 0x7f9aa80065d0 con 0x7f9aa8009140 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9aa8009e80 con 0x7f9aa8009140 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9ab4003d90 con 0x7f9aa8009140 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient: dump: 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:27.985 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2310617361 0 0) 0x7f9aa8009e80 con 0x7f9aa8009140 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9aa800c400 con 0x7f9aa8009140 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3972 (unknown 0 0 1808542598) 0x7f9ab40059e0 con 0x7f9aa8009140 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 >> v1:172.21.15.78:6789/0 conn(0x7f9aa8009140 legacy=0x7f9aa80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.966+0000 7f9abb7fe640 1 -- 172.21.15.78:0/1399428205 >> v1:172.21.15.78:6789/0 conn(0x7f9aa8009140 legacy=0x7f9aa80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 32, 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17735, 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:27.987 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:27.988 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:27.989 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:39:27.990 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:27.991 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:27.992 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:27.993 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:27.994 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:27.995 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:27.996 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:39:27.997 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:39:27.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 10 librados: watch_flush enter 2024-09-15T20:39:27.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 10 librados: watch_flush exit 2024-09-15T20:39:27.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 20 client.84485.objecter shutdown clearing up homeless session... 2024-09-15T20:39:27.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 10 client.84485.objecter successfully canceled tick 2024-09-15T20:39:27.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 >> v1:172.21.15.146:6812/435598642 conn(0x7f9aa403e240 legacy=0x7f9aa4040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:27.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 10 monclient: shutdown 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 >> v1:172.21.15.78:6790/0 conn(0x7f9ac409f5a0 legacy=0x7f9ac411ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac359e640 1 -- 172.21.15.78:0/1399428205 reap_dead start 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac359e640 1 -- 172.21.15.78:0/1399428205 reap_dead start 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 shutdown_connections 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 >> 172.21.15.78:0/1399428205 conn(0x7f9ac40a63c0 msgr2=0x7f9ac40a4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 shutdown_connections 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 1 -- 172.21.15.78:0/1399428205 wait complete. 2024-09-15T20:39:28.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:27.970+0000 7f9ac90e9640 1 librados: shutdown 2024-09-15T20:39:28.000 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:39:28.000 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:39:28.001 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:39:28.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient: get_monmap_and_config 2024-09-15T20:39:28.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient: build_initial_monmap 2024-09-15T20:39:28.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient: monmap: 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:28.117699+0000 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:28.117699+0000 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:28.117 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 1 Processor -- start 2024-09-15T20:39:28.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 1 -- start start 2024-09-15T20:39:28.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient: init 2024-09-15T20:39:28.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient(hunting): picked mon.noname-b con 0x7f5494135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient(hunting): picked mon.noname-c con 0x7f54940a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient(hunting): picked mon.noname-a con 0x7f549413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54940745e0 con 0x7f549413c220 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549406ecf0 con 0x7f54940a24e0 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549406efd0 con 0x7f5494135b60 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient(hunting): authenticate will time out at 1629.002475s 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54937fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f549413c220 0x7f549413e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49490/0 (socket says 172.21.15.78:49490) 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54937fe640 1 -- 172.21.15.78:0/624449671 learned_addr learned my addr 172.21.15.78:0/624449671 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:28.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f5492ffd640 1 --1- 172.21.15.78:0/624449671 >> v1:172.21.15.78:6790/0 conn(0x7f5494135b60 0x7f5494135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51198/0 (socket says 172.21.15.78:51198) 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400122937 0 0) 0x7f549406efd0 con 0x7f5494135b60 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient(hunting): my global_id is 84488 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5480003530 con 0x7f5494135b60 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2180900026 0 0) 0x7f54940745e0 con 0x7f549413c220 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient(hunting): my global_id is 104151 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f549406efd0 con 0x7f549413c220 2024-09-15T20:39:28.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2733334622 0 0) 0x7f5480003530 con 0x7f5494135b60 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 >> v1:172.21.15.146:6789/0 conn(0x7f54940a24e0 legacy=0x7f549413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 >> v1:172.21.15.78:6789/0 conn(0x7f549413c220 legacy=0x7f549413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: _finish_hunting 0 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 monclient: found mon.noname-b 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5494136650 con 0x7f5494135b60 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: _finish_auth 0 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: _check_auth_tickets 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f54840016a0 con 0x7f5494135b60 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:28.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: dump: 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 5 monclient: authenticate success, global_id 84488 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:28.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5484001840 con 0x7f5494135b60 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 1 -- 172.21.15.78:0/624449671 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5484001c10 con 0x7f5494135b60 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54927fc640 10 monclient: dump: 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.125 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 4 set_mon_vals no callback set 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 10 monclient: shutdown 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.114+0000 7f54993ae640 1 -- 172.21.15.78:0/624449671 >> v1:172.21.15.78:6790/0 conn(0x7f5494135b60 legacy=0x7f5494135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- 172.21.15.78:0/624449671 shutdown_connections 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- 172.21.15.78:0/624449671 >> 172.21.15.78:0/624449671 conn(0x7f54940a63c0 msgr2=0x7f54940a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- 172.21.15.78:0/624449671 shutdown_connections 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- 172.21.15.78:0/624449671 wait complete. 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: build_initial_monmap 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: monmap: 2024-09-15T20:39:28.126 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 librados: starting msgr at 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 librados: starting objecter 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 Processor -- start 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- start start 2024-09-15T20:39:28.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 librados: setting wanted keys 2024-09-15T20:39:28.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 librados: calling monclient init 2024-09-15T20:39:28.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: init 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient(hunting): picked mon.noname-c con 0x7f54940a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient(hunting): picked mon.noname-b con 0x7f5494135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient(hunting): picked mon.noname-a con 0x7f549413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5494135570 con 0x7f549413c220 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549413b9b0 con 0x7f5494135b60 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f549409d180 con 0x7f54940a24e0 2024-09-15T20:39:28.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient(hunting): authenticate will time out at 1629.005578s 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f5492ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f54940a24e0 0x7f5494138120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51212/0 (socket says 172.21.15.78:51212) 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f5492ffd640 1 -- 172.21.15.78:0/584250819 learned_addr learned my addr 172.21.15.78:0/584250819 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.?.objecter ms_handle_connect 0x7f549413c220 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.?.objecter ms_handle_connect 0x7f54940a24e0 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1636550108 0 0) 0x7f5494135570 con 0x7f549413c220 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): my global_id is 104154 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5474004ec0 con 0x7f549413c220 2024-09-15T20:39:28.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 482596757 0 0) 0x7f549409d180 con 0x7f54940a24e0 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): my global_id is 84491 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5494135570 con 0x7f54940a24e0 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.?.objecter ms_handle_connect 0x7f5494135b60 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2156080401 0 0) 0x7f5474004ec0 con 0x7f549413c220 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f549409d180 con 0x7f549413c220 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f548c001c40 con 0x7f549413c220 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:39:28.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): dump: 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1395806379 0 0) 0x7f549413b9b0 con 0x7f5494135b60 2024-09-15T20:39:28.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): my global_id is 23548 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5474004ec0 con 0x7f5494135b60 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3920392640 0 0) 0x7f5494135570 con 0x7f54940a24e0 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f549413b9b0 con 0x7f54940a24e0 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5484001b90 con 0x7f54940a24e0 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient(hunting): dump: 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.137 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1909206721 0 0) 0x7f549409d180 con 0x7f549413c220 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 >> v1:172.21.15.78:6790/0 conn(0x7f54940a24e0 legacy=0x7f5494138120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 >> v1:172.21.15.146:6789/0 conn(0x7f5494135b60 legacy=0x7f5494138830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 monclient: found mon.a 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f549413b0d0 con 0x7f549413c220 2024-09-15T20:39:28.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient: _finish_auth 0 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 5 monclient: authenticate success, global_id 104154 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: _renew_subs 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5494139be0 con 0x7f549413c220 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 client.104154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: _renew_subs 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5494138f40 con 0x7f549413c220 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 librados: init done 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f548c004b70 con 0x7f549413c220 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:28.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f548c004ec0 con 0x7f549413c220 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 monclient: dump: 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.140 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f5491ffb640 4 set_mon_vals no callback set 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f54780065d0 con 0x7f5478002870 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.104154.objecter ms_handle_connect 0x7f5478002870 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.104154.objecter resend_mon_ops 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f548c010f30 con 0x7f549413c220 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(341..341 src has 1..341) ==== 5333+0+0 (unknown 208272338 0 0) 0x7f548c04ba80 con 0x7f549413c220 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 10 client.104154.objecter ms_dispatch 0x7f5494000ba0 osd_map(341..341 src has 1..341) 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 3 client.104154.objecter handle_osd_map got epochs [341,341] > 0 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 3 client.104154.objecter handle_osd_map decoding full epoch 341 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 20 client.104154.objecter dump_active .. 0 homeless 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.118+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1360503668 0 0) 0x7f54780065d0 con 0x7f5478002870 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient(hunting): my global_id is 84494 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f549409d180 con 0x7f5478002870 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1092484916 0 0) 0x7f549409d180 con 0x7f5478002870 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f54780065d0 con 0x7f5478002870 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5484004a80 con 0x7f5478002870 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548a7fc640 10 monclient: get_auth_request con 0x7f547403e240 auth_method 0 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient: dump: 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.142 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1631255222 0 0) 0x7f54780065d0 con 0x7f5478002870 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f547800a170 con 0x7f5478002870 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 client.104154.objecter ms_handle_connect 0x7f547403e240 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5484004a80 con 0x7f5478002870 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 >> v1:172.21.15.78:6790/0 conn(0x7f5478002870 legacy=0x7f5478004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.122+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 >> v1:172.21.15.78:6790/0 conn(0x7f5478002870 legacy=0x7f5478004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:28.143 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f54993ae640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5478009e30 con 0x7f54780090f0 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 client.104154.objecter ms_handle_connect 0x7f54780090f0 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 client.104154.objecter resend_mon_ops 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 106843938 0 0) 0x7f5478009e30 con 0x7f54780090f0 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient(hunting): my global_id is 84497 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f54780065d0 con 0x7f54780090f0 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1046594000 0 0) 0x7f54780065d0 con 0x7f54780090f0 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5478009e30 con 0x7f54780090f0 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5484005080 con 0x7f54780090f0 2024-09-15T20:39:28.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient: dump: 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3834370261 0 0) 0x7f5478009e30 con 0x7f54780090f0 2024-09-15T20:39:28.145 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f547800c400 con 0x7f54780090f0 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 3907713097) 0x7f5484007300 con 0x7f54780090f0 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 >> v1:172.21.15.78:6790/0 conn(0x7f54780090f0 legacy=0x7f5478009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.126+0000 7f548bfff640 1 -- 172.21.15.78:0/584250819 >> v1:172.21.15.78:6790/0 conn(0x7f54780090f0 legacy=0x7f5478009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:39:28.146 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 32, 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 239813, 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:28.147 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:28.148 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:39:28.149 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:28.150 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.151 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.152 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.153 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:28.154 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:28.155 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.156 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:39:28.157 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:39:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 10 librados: watch_flush enter 2024-09-15T20:39:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 10 librados: watch_flush exit 2024-09-15T20:39:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 20 client.104154.objecter shutdown clearing up homeless session... 2024-09-15T20:39:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 10 client.104154.objecter successfully canceled tick 2024-09-15T20:39:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 >> v1:172.21.15.146:6812/435598642 conn(0x7f547403e240 legacy=0x7f5474040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 10 monclient: shutdown 2024-09-15T20:39:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:28.159 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 >> v1:172.21.15.78:6789/0 conn(0x7f549413c220 legacy=0x7f54941d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54937fe640 1 -- 172.21.15.78:0/584250819 reap_dead start 2024-09-15T20:39:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54937fe640 1 -- 172.21.15.78:0/584250819 reap_dead start 2024-09-15T20:39:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 shutdown_connections 2024-09-15T20:39:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 >> 172.21.15.78:0/584250819 conn(0x7f54940a63c0 msgr2=0x7f549413f440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 shutdown_connections 2024-09-15T20:39:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 1 -- 172.21.15.78:0/584250819 wait complete. 2024-09-15T20:39:28.160 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.130+0000 7f54993ae640 1 librados: shutdown 2024-09-15T20:39:28.160 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:39:28.160 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:39:28.161 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:39:28.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient: get_monmap_and_config 2024-09-15T20:39:28.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient: build_initial_monmap 2024-09-15T20:39:28.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient: monmap: 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:28.273964+0000 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:28.273964+0000 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:28.273 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 1 Processor -- start 2024-09-15T20:39:28.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 1 -- start start 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient: init 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient(hunting): picked mon.noname-b con 0x7f95ac137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient(hunting): picked mon.noname-c con 0x7f95ac0a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient(hunting): picked mon.noname-a con 0x7f95ac13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ac0a6420 con 0x7f95ac13e5f0 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ac0a6700 con 0x7f95ac0a9610 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ac0a2110 con 0x7f95ac137ce0 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient(hunting): authenticate will time out at 1629.158662s 2024-09-15T20:39:28.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b2143640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f95ac137ce0 0x7f95ac1380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51236/0 (socket says 172.21.15.78:51236) 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b2143640 1 -- 172.21.15.78:0/407703293 learned_addr learned my addr 172.21.15.78:0/407703293 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336953820 0 0) 0x7f95ac0a6420 con 0x7f95ac13e5f0 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient(hunting): my global_id is 104157 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9598003530 con 0x7f95ac13e5f0 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2037431556 0 0) 0x7f9598003530 con 0x7f95ac13e5f0 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 >> v1:172.21.15.78:6790/0 conn(0x7f95ac137ce0 legacy=0x7f95ac1380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 >> v1:172.21.15.146:6789/0 conn(0x7f95ac0a9610 legacy=0x7f95ac13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: _finish_hunting 0 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 monclient: found mon.noname-a 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:28.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ac0a4bc0 con 0x7f95ac13e5f0 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: _finish_auth 0 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: _check_auth_tickets 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95a4001a20 con 0x7f95ac13e5f0 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: dump: 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.278 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 5 monclient: authenticate success, global_id 104157 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95a4001c40 con 0x7f95ac13e5f0 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 1 -- 172.21.15.78:0/407703293 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95a4002050 con 0x7f95ac13e5f0 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b1141640 10 monclient: dump: 2024-09-15T20:39:28.279 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.270+0000 7f95b43a7640 4 set_mon_vals no callback set 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient: shutdown 2024-09-15T20:39:28.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:28.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- 172.21.15.78:0/407703293 >> v1:172.21.15.78:6789/0 conn(0x7f95ac13e5f0 legacy=0x7f95ac1409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- 172.21.15.78:0/407703293 shutdown_connections 2024-09-15T20:39:28.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- 172.21.15.78:0/407703293 >> 172.21.15.78:0/407703293 conn(0x7f95ac0aa6c0 msgr2=0x7f95ac0aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:28.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- 172.21.15.78:0/407703293 shutdown_connections 2024-09-15T20:39:28.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- 172.21.15.78:0/407703293 wait complete. 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient: build_initial_monmap 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient: monmap: 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:28.284 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 librados: starting msgr at 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 librados: starting objecter 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 Processor -- start 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- start start 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 librados: setting wanted keys 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 librados: calling monclient init 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient: init 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient(hunting): picked mon.noname-b con 0x7f95ac0a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:28.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient(hunting): picked mon.noname-c con 0x7f95ac137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient(hunting): picked mon.noname-a con 0x7f95ac13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ac1372b0 con 0x7f95ac13e5f0 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ac09e530 con 0x7f95ac0a9610 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95ac09fff0 con 0x7f95ac137ce0 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b43a7640 10 monclient(hunting): authenticate will time out at 1629.162035s 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b1942640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f95ac137ce0 0x7f95ac09d400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51252/0 (socket says 172.21.15.78:51252) 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95b1942640 1 -- 172.21.15.78:0/2816716251 learned_addr learned my addr 172.21.15.78:0/2816716251 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95a2ffd640 10 client.?.objecter ms_handle_connect 0x7f95ac13e5f0 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95a2ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95a2ffd640 10 client.?.objecter ms_handle_connect 0x7f95ac137ce0 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95a2ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:28.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95a2ffd640 10 client.?.objecter ms_handle_connect 0x7f95ac0a9610 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.274+0000 7f95a2ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4056014100 0 0) 0x7f95ac09fff0 con 0x7f95ac137ce0 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): my global_id is 84500 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9590004ec0 con 0x7f95ac137ce0 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 889368850 0 0) 0x7f95ac1372b0 con 0x7f95ac13e5f0 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): my global_id is 104160 2024-09-15T20:39:28.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95ac09fff0 con 0x7f95ac13e5f0 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 260817165 0 0) 0x7f95ac09fff0 con 0x7f95ac13e5f0 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95ac1372b0 con 0x7f95ac13e5f0 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95a4001f90 con 0x7f95ac13e5f0 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): dump: 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.288 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 625806001 0 0) 0x7f95ac1372b0 con 0x7f95ac13e5f0 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 >> v1:172.21.15.78:6790/0 conn(0x7f95ac137ce0 legacy=0x7f95ac09d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 >> v1:172.21.15.146:6789/0 conn(0x7f95ac0a9610 legacy=0x7f95ac09ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 monclient: found mon.a 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95ac09f090 con 0x7f95ac13e5f0 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient: _finish_auth 0 2024-09-15T20:39:28.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 5 monclient: authenticate success, global_id 104160 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 10 monclient: _renew_subs 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f95ac13de60 con 0x7f95ac13e5f0 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 10 client.104160.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 10 monclient: _renew_subs 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f95ac139340 con 0x7f95ac13e5f0 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 1 librados: init done 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95a4001ba0 con 0x7f95ac13e5f0 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:28.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95a4003e90 con 0x7f95ac13e5f0 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 monclient: dump: 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b0940640 4 set_mon_vals no callback set 2024-09-15T20:39:28.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:28.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f95940065d0 con 0x7f9594002870 2024-09-15T20:39:28.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f95a400ff00 con 0x7f95ac13e5f0 2024-09-15T20:39:28.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(342..342 src has 1..342) ==== 5346+0+0 (unknown 1453557862 0 0) 0x7f95a404a860 con 0x7f95ac13e5f0 2024-09-15T20:39:28.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 10 client.104160.objecter ms_dispatch 0x7f95ac000ba0 osd_map(342..342 src has 1..342) 2024-09-15T20:39:28.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 3 client.104160.objecter handle_osd_map got epochs [342,342] > 0 2024-09-15T20:39:28.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 3 client.104160.objecter handle_osd_map decoding full epoch 342 2024-09-15T20:39:28.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.278+0000 7f95a2ffd640 20 client.104160.objecter dump_active .. 0 homeless 2024-09-15T20:39:28.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.282+0000 7f95b1942640 10 monclient: get_auth_request con 0x7f959003e140 auth_method 0 2024-09-15T20:39:28.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.282+0000 7f95a2ffd640 10 client.104160.objecter ms_handle_connect 0x7f9594002870 2024-09-15T20:39:28.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.282+0000 7f95a2ffd640 10 client.104160.objecter resend_mon_ops 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.282+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1882441882 0 0) 0x7f95940065d0 con 0x7f9594002870 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 10 monclient(hunting): my global_id is 23551 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95ac1372b0 con 0x7f9594002870 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 229262848 0 0) 0x7f95ac1372b0 con 0x7f9594002870 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95940065d0 con 0x7f9594002870 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95a8001680 con 0x7f9594002870 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 10 monclient: dump: 2024-09-15T20:39:28.294 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.286+0000 7f95a2ffd640 10 client.104160.objecter ms_handle_connect 0x7f959003e140 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.290+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 760442052 0 0) 0x7f95940065d0 con 0x7f9594002870 2024-09-15T20:39:28.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.290+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f959400a170 con 0x7f9594002870 2024-09-15T20:39:28.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.298+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f95a8001880 con 0x7f9594002870 2024-09-15T20:39:28.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.298+0000 7f95a2ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:28.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.298+0000 7f95a2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:28.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.298+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 >> v1:172.21.15.146:6789/0 conn(0x7f9594002870 legacy=0x7f9594004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.298+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 >> v1:172.21.15.146:6789/0 conn(0x7f9594002870 legacy=0x7f9594004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:28.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.306+0000 7f95b43a7640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:39:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.306+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9594009e60 con 0x7f9594009120 2024-09-15T20:39:28.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.306+0000 7f95a2ffd640 10 client.104160.objecter ms_handle_connect 0x7f9594009120 2024-09-15T20:39:28.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.306+0000 7f95a2ffd640 10 client.104160.objecter resend_mon_ops 2024-09-15T20:39:28.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253946524 0 0) 0x7f9594009e60 con 0x7f9594009120 2024-09-15T20:39:28.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient(hunting): my global_id is 23554 2024-09-15T20:39:28.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95940065d0 con 0x7f9594009120 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3393259477 0 0) 0x7f95940065d0 con 0x7f9594009120 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9594009e60 con 0x7f9594009120 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95a8004990 con 0x7f9594009120 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient: dump: 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.314 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.315 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.315 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.315 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.315 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.315 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3729460683 0 0) 0x7f9594009e60 con 0x7f9594009120 2024-09-15T20:39:28.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f959400c400 con 0x7f9594009120 2024-09-15T20:39:28.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 294278332) 0x7f95a80058e0 con 0x7f9594009120 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 >> v1:172.21.15.146:6789/0 conn(0x7f9594009120 legacy=0x7f95940095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95a2ffd640 1 -- 172.21.15.78:0/2816716251 >> v1:172.21.15.146:6789/0 conn(0x7f9594009120 legacy=0x7f95940095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:39:28.316 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 32, 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 480818, 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 3, 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:28.317 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:28.318 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:39:28.319 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:39:28.320 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:28.321 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:28.322 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:28.323 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:28.324 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.325 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:28.326 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:28.327 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:39:28.328 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:39:28.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b43a7640 10 librados: watch_flush enter 2024-09-15T20:39:28.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b43a7640 10 librados: watch_flush exit 2024-09-15T20:39:28.330 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b43a7640 20 client.104160.objecter shutdown clearing up homeless session... 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b43a7640 10 client.104160.objecter successfully canceled tick 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 >> v1:172.21.15.146:6812/435598642 conn(0x7f959003e140 legacy=0x7f9590040600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b43a7640 10 monclient: shutdown 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b43a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 >> v1:172.21.15.78:6789/0 conn(0x7f95ac13e5f0 legacy=0x7f95ac1d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.310+0000 7f95b2944640 1 -- 172.21.15.78:0/2816716251 reap_dead start 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.314+0000 7f95b2944640 1 -- 172.21.15.78:0/2816716251 reap_dead start 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.314+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 shutdown_connections 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.314+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 >> 172.21.15.78:0/2816716251 conn(0x7f95ac0aa6c0 msgr2=0x7f95ac0a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.314+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 shutdown_connections 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.314+0000 7f95b43a7640 1 -- 172.21.15.78:0/2816716251 wait complete. 2024-09-15T20:39:28.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.314+0000 7f95b43a7640 1 librados: shutdown 2024-09-15T20:39:28.332 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:39:28.332 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T20:39:28.332 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T20:39:28.332 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T20:39:28.332 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T20:39:28.332 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T20:39:28.332 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:39:28.342 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T20:39:28.342 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:39:28.343 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:39:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient: get_monmap_and_config 2024-09-15T20:39:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient: build_initial_monmap 2024-09-15T20:39:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient: monmap: 2024-09-15T20:39:28.501 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:28.502339+0000 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:28.502339+0000 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:28.502 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 1 Processor -- start 2024-09-15T20:39:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 1 -- start start 2024-09-15T20:39:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient: init 2024-09-15T20:39:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:39:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient(hunting): picked mon.noname-c con 0x7f29f0135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:28.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient(hunting): picked mon.noname-a con 0x7f29f00a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient(hunting): picked mon.noname-b con 0x7f29f013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f00745e0 con 0x7f29f00a24e0 2024-09-15T20:39:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f006ecf0 con 0x7f29f0135b60 2024-09-15T20:39:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f006efd0 con 0x7f29f013c220 2024-09-15T20:39:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f831f640 10 monclient(hunting): authenticate will time out at 1629.387336s 2024-09-15T20:39:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f58ba640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f29f00a24e0 legacy=0x7f29f013ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:28.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f68bc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f29f013c220 0x7f29f013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51264/0 (socket says 172.21.15.78:51264) 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f68bc640 1 -- 172.21.15.78:0/48460088 learned_addr learned my addr 172.21.15.78:0/48460088 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1255730280 0 0) 0x7f29f006efd0 con 0x7f29f013c220 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f50b9640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f50b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f50b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f50b9640 10 monclient(hunting): my global_id is 84503 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.498+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29dc003530 con 0x7f29f013c220 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1435201565 0 0) 0x7f29f006ecf0 con 0x7f29f0135b60 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient(hunting): my global_id is 23557 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29f006efd0 con 0x7f29f0135b60 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3783112478 0 0) 0x7f29dc003530 con 0x7f29f013c220 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 >> v1:172.21.15.146:6789/0 conn(0x7f29f0135b60 legacy=0x7f29f0135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 >> v1:172.21.15.78:6789/0 conn(0x7f29f00a24e0 legacy=0x7f29f013ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: _finish_hunting 0 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 monclient: found mon.noname-b 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:28.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f0136650 con 0x7f29f013c220 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: _finish_auth 0 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: _check_auth_tickets 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f29e8001a20 con 0x7f29f013c220 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: dump: 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.514 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 5 monclient: authenticate success, global_id 84503 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29e8001bc0 con 0x7f29f013c220 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:28.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 1 -- 172.21.15.78:0/48460088 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f29e8001f90 con 0x7f29f013c220 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f50b9640 10 monclient: dump: 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.516 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 4 set_mon_vals no callback set 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: shutdown 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- 172.21.15.78:0/48460088 >> v1:172.21.15.78:6790/0 conn(0x7f29f013c220 legacy=0x7f29f013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- 172.21.15.78:0/48460088 shutdown_connections 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- 172.21.15.78:0/48460088 >> 172.21.15.78:0/48460088 conn(0x7f29f00a63c0 msgr2=0x7f29f00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- 172.21.15.78:0/48460088 shutdown_connections 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- 172.21.15.78:0/48460088 wait complete. 2024-09-15T20:39:28.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: build_initial_monmap 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: monmap: 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 librados: starting msgr at 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 librados: starting objecter 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 Processor -- start 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- start start 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 librados: setting wanted keys 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 librados: calling monclient init 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: init 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient(hunting): picked mon.noname-b con 0x7f29f00a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient(hunting): picked mon.noname-a con 0x7f29f0135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:28.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient(hunting): picked mon.noname-c con 0x7f29f013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f013b590 con 0x7f29f0135b60 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f0138670 con 0x7f29f00a24e0 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29f013b8f0 con 0x7f29f013c220 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient(hunting): authenticate will time out at 1629.390501s 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f58ba640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f29f0135b60 legacy=0x7f29f0072400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f68bc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f29f013c220 0x7f29f01d2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51280/0 (socket says 172.21.15.78:51280) 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f68bc640 1 -- 172.21.15.78:0/1610117294 learned_addr learned my addr 172.21.15.78:0/1610117294 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 client.?.objecter ms_handle_connect 0x7f29f013c220 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:28.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 client.?.objecter ms_handle_connect 0x7f29f00a24e0 2024-09-15T20:39:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144910831 0 0) 0x7f29f013b8f0 con 0x7f29f013c220 2024-09-15T20:39:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): my global_id is 84506 2024-09-15T20:39:28.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29d4004ec0 con 0x7f29f013c220 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491341259 0 0) 0x7f29f0138670 con 0x7f29f00a24e0 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): my global_id is 23560 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f29f013b8f0 con 0x7f29f00a24e0 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 36319415 0 0) 0x7f29d4004ec0 con 0x7f29f013c220 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f29f0138670 con 0x7f29f013c220 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f29e8001f10 con 0x7f29f013c220 2024-09-15T20:39:28.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): dump: 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.523 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1281093012 0 0) 0x7f29f0138670 con 0x7f29f013c220 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 >> v1:172.21.15.146:6789/0 conn(0x7f29f00a24e0 legacy=0x7f29f0071cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 >> v1:172.21.15.78:6789/0 conn(0x7f29f0135b60 legacy=0x7f29f0072400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 monclient: found mon.c 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29f009d180 con 0x7f29f013c220 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient: _finish_auth 0 2024-09-15T20:39:28.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 5 monclient: authenticate success, global_id 84506 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: _renew_subs 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f29f0135060 con 0x7f29f013c220 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 client.84506.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: _renew_subs 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f29f0138390 con 0x7f29f013c220 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f831f640 1 librados: init done 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f29e8001ba0 con 0x7f29f013c220 2024-09-15T20:39:28.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f29e8004f80 con 0x7f29f013c220 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29f48b8640 4 set_mon_vals no callback set 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.502+0000 7f29e6ffd640 10 monclient: dump: 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:28.526 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29f831f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29f831f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29f831f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d8005090 con 0x7f29f013c220 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f29e8010ff0 con 0x7f29f013c220 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(342..342 src has 1..342) ==== 5346+0+0 (unknown 1453557862 0 0) 0x7f29e804ba60 con 0x7f29f013c220 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29e6ffd640 10 client.84506.objecter ms_dispatch 0x7f29f0000ba0 osd_map(342..342 src has 1..342) 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29e6ffd640 3 client.84506.objecter handle_osd_map got epochs [342,342] > 0 2024-09-15T20:39:28.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29e6ffd640 3 client.84506.objecter handle_osd_map decoding full epoch 342 2024-09-15T20:39:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29e6ffd640 20 client.84506.objecter dump_active .. 0 homeless 2024-09-15T20:39:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29f60bb640 10 monclient: get_auth_request con 0x7f29d403e0d0 auth_method 0 2024-09-15T20:39:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.506+0000 7f29e6ffd640 10 client.84506.objecter ms_handle_connect 0x7f29d403e0d0 2024-09-15T20:39:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.510+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f29e8016dc0 con 0x7f29f013c220 2024-09-15T20:39:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.510+0000 7f29e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:28.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.510+0000 7f29e6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.734+0000 7f29f831f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:39:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.734+0000 7f29f831f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.734+0000 7f29f831f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:28.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.734+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f29d8005380 con 0x7f29f013c220 2024-09-15T20:39:28.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.734+0000 7f29e6ffd640 1 -- 172.21.15.78:0/1610117294 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2839893239) 0x7f29e80166c0 con 0x7f29f013c220 2024-09-15T20:39:28.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.734+0000 7f29e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:28.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.734+0000 7f29e6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:28.739 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 10 librados: watch_flush enter 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 10 librados: watch_flush exit 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 20 client.84506.objecter shutdown clearing up homeless session... 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 10 client.84506.objecter successfully canceled tick 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 >> v1:172.21.15.146:6812/435598642 conn(0x7f29d403e0d0 legacy=0x7f29d4040590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 10 monclient: shutdown 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 >> v1:172.21.15.78:6790/0 conn(0x7f29f013c220 legacy=0x7f29f01d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 shutdown_connections 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 >> 172.21.15.78:0/1610117294 conn(0x7f29f00a63c0 msgr2=0x7f29f00a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 shutdown_connections 2024-09-15T20:39:28.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 1 -- 172.21.15.78:0/1610117294 wait complete. 2024-09-15T20:39:28.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:28.738+0000 7f29f831f640 1 librados: shutdown 2024-09-15T20:39:31.756 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:39:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient: get_monmap_and_config 2024-09-15T20:39:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient: build_initial_monmap 2024-09-15T20:39:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient: monmap: 2024-09-15T20:39:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:31.886446+0000 2024-09-15T20:39:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:31.886446+0000 2024-09-15T20:39:31.885 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:31.886 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:31.886 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:31.886 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:31.886 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:31.886 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:31.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 1 Processor -- start 2024-09-15T20:39:31.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 1 -- start start 2024-09-15T20:39:31.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient: init 2024-09-15T20:39:31.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient(hunting): picked mon.noname-a con 0x7f3744135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient(hunting): picked mon.noname-c con 0x7f37440a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient(hunting): picked mon.noname-b con 0x7f374413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37440745e0 con 0x7f3744135b60 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374406ecf0 con 0x7f37440a24e0 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3742ffd640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f3744135b60 legacy=0x7f3744135f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374406efd0 con 0x7f374413c220 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient(hunting): authenticate will time out at 1632.771056s 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f37437fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f374413c220 0x7f374413e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51396/0 (socket says 172.21.15.78:51396) 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f37437fe640 1 -- 172.21.15.78:0/1948549933 learned_addr learned my addr 172.21.15.78:0/1948549933 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:31.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1425084694 0 0) 0x7f374406efd0 con 0x7f374413c220 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient(hunting): my global_id is 84509 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3730003530 con 0x7f374413c220 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4252561684 0 0) 0x7f3730003530 con 0x7f374413c220 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 >> v1:172.21.15.146:6789/0 conn(0x7f37440a24e0 legacy=0x7f374413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 >> v1:172.21.15.78:6789/0 conn(0x7f3744135b60 legacy=0x7f3744135f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 monclient: found mon.noname-b 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:31.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3744136650 con 0x7f374413c220 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: _finish_auth 0 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f373c0016a0 con 0x7f374413c220 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 5 monclient: authenticate success, global_id 84509 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: dump: 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:31.896 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f373c001900 con 0x7f374413c220 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:31.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f374955c640 4 set_mon_vals no callback set 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 1 -- 172.21.15.78:0/1948549933 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f373c001d10 con 0x7f374413c220 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.882+0000 7f3741ffb640 10 monclient: dump: 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:31.898 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: shutdown 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- 172.21.15.78:0/1948549933 >> v1:172.21.15.78:6790/0 conn(0x7f374413c220 legacy=0x7f374413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- 172.21.15.78:0/1948549933 shutdown_connections 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- 172.21.15.78:0/1948549933 >> 172.21.15.78:0/1948549933 conn(0x7f37440a63c0 msgr2=0x7f37440a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- 172.21.15.78:0/1948549933 shutdown_connections 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- 172.21.15.78:0/1948549933 wait complete. 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: build_initial_monmap 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: monmap: 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:31.899 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 librados: starting msgr at 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 librados: starting objecter 2024-09-15T20:39:31.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 Processor -- start 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- start start 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 librados: setting wanted keys 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 librados: calling monclient init 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: init 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient(hunting): picked mon.noname-b con 0x7f37440a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient(hunting): picked mon.noname-a con 0x7f3744135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient(hunting): picked mon.noname-c con 0x7f374413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3744139d50 con 0x7f3744135b60 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374413a030 con 0x7f37440a24e0 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37441376f0 con 0x7f374413c220 2024-09-15T20:39:31.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient(hunting): authenticate will time out at 1632.774480s 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f37427fc640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f3744135b60 legacy=0x7f3744136fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f37437fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f374413c220 0x7f37441d2cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51412/0 (socket says 172.21.15.78:51412) 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f37437fe640 1 -- 172.21.15.78:0/1295996887 learned_addr learned my addr 172.21.15.78:0/1295996887 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 client.?.objecter ms_handle_connect 0x7f374413c220 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492225676 0 0) 0x7f37441376f0 con 0x7f374413c220 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient(hunting): my global_id is 84512 2024-09-15T20:39:31.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3724004ef0 con 0x7f374413c220 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2635879677 0 0) 0x7f3724004ef0 con 0x7f374413c220 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37441376f0 con 0x7f374413c220 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f373c0014c0 con 0x7f374413c220 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient(hunting): dump: 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:31.903 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 client.?.objecter ms_handle_connect 0x7f37440a24e0 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3206928629 0 0) 0x7f37441376f0 con 0x7f374413c220 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 >> v1:172.21.15.146:6789/0 conn(0x7f37440a24e0 legacy=0x7f37441368d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 >> v1:172.21.15.78:6789/0 conn(0x7f3744135b60 legacy=0x7f3744136fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:39:31.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 monclient: found mon.c 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374409d180 con 0x7f374413c220 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient: _finish_auth 0 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 5 monclient: authenticate success, global_id 84512 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: _renew_subs 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f374413a5f0 con 0x7f374413c220 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 client.84512.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: _renew_subs 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:31.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f374413b510 con 0x7f374413c220 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f374955c640 1 librados: init done 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f373c004c80 con 0x7f374413c220 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f3740ff9640 4 set_mon_vals no callback set 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f373c004e30 con 0x7f374413c220 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.886+0000 7f373b7fe640 10 monclient: dump: 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:31.906 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f373c010ea0 con 0x7f374413c220 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(342..342 src has 1..342) ==== 5346+0+0 (unknown 1453557862 0 0) 0x7f373c04b8d0 con 0x7f374413c220 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f373b7fe640 10 client.84512.objecter ms_dispatch 0x7f3744000ba0 osd_map(342..342 src has 1..342) 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f373b7fe640 3 client.84512.objecter handle_osd_map got epochs [342,342] > 0 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f373b7fe640 3 client.84512.objecter handle_osd_map decoding full epoch 342 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f374955c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:31.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f374955c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:31.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f374955c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:31.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f373b7fe640 20 client.84512.objecter dump_active .. 0 homeless 2024-09-15T20:39:31.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3728005090 con 0x7f374413c220 2024-09-15T20:39:31.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f3742ffd640 10 monclient: get_auth_request con 0x7f372403e0d0 auth_method 0 2024-09-15T20:39:31.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.890+0000 7f373b7fe640 10 client.84512.objecter ms_handle_connect 0x7f372403e0d0 2024-09-15T20:39:31.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.894+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f373c016c30 con 0x7f374413c220 2024-09-15T20:39:31.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.894+0000 7f373b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:31.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:31.894+0000 7f373b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:32.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f374955c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:39:32.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f374955c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:32.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f374955c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:32.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f3728005380 con 0x7f374413c220 2024-09-15T20:39:32.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f373b7fe640 1 -- 172.21.15.78:0/1295996887 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2933601774) 0x7f373c016530 con 0x7f374413c220 2024-09-15T20:39:32.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f373b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:32.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f373b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:32.118 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":32,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f374955c640 10 librados: watch_flush enter 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f374955c640 10 librados: watch_flush exit 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f374955c640 20 client.84512.objecter shutdown clearing up homeless session... 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.114+0000 7f374955c640 10 client.84512.objecter successfully canceled tick 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 >> v1:172.21.15.146:6812/435598642 conn(0x7f372403e0d0 legacy=0x7f3724040590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 10 monclient: shutdown 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 >> v1:172.21.15.78:6790/0 conn(0x7f374413c220 legacy=0x7f37441d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 shutdown_connections 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 >> 172.21.15.78:0/1295996887 conn(0x7f37440a63c0 msgr2=0x7f37440a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 shutdown_connections 2024-09-15T20:39:32.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 1 -- 172.21.15.78:0/1295996887 wait complete. 2024-09-15T20:39:32.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:32.118+0000 7f374955c640 1 librados: shutdown 2024-09-15T20:39:35.136 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.258+0000 7fb8592bc640 10 monclient: get_monmap_and_config 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.258+0000 7fb8592bc640 10 monclient: build_initial_monmap 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.258+0000 7fb8592bc640 10 monclient: monmap: 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:35.264322+0000 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:35.264322+0000 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:35.264 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:35.264 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:35.264 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:35.264 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:35.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 1 Processor -- start 2024-09-15T20:39:35.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 1 -- start start 2024-09-15T20:39:35.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 10 monclient: init 2024-09-15T20:39:35.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:35.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:39:35.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 10 monclient(hunting): picked mon.noname-b con 0x7fb854135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:35.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 10 monclient(hunting): picked mon.noname-a con 0x7fb8540a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:35.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 10 monclient(hunting): picked mon.noname-c con 0x7fb85413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8540745e0 con 0x7fb8540a24e0 2024-09-15T20:39:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85406ecf0 con 0x7fb85413c220 2024-09-15T20:39:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85406efd0 con 0x7fb854135b60 2024-09-15T20:39:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb8592bc640 10 monclient(hunting): authenticate will time out at 1636.148836s 2024-09-15T20:39:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb84a7fc640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb85413ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb852ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb854135b60 0x7fb854135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51414/0 (socket says 172.21.15.78:51414) 2024-09-15T20:39:35.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.262+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 learned_addr learned my addr 172.21.15.78:0/1036603742 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:35.469 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.466+0000 7fb84a7fc640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb85413ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:35.870+0000 7fb84a7fc640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb85413ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:36.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:36.674+0000 7fb84a7fc640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb85413ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 10 monclient(hunting): tick 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 1 monclient(hunting): continuing hunt 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6790/0 conn(0x7fb854135b60 legacy=0x7fb854135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.146:6789/0 conn(0x7fb85413c220 legacy=0x7fb85413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb85413ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb8540a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb854135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:38.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb85413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:38.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb854136650 con 0x7fb8540a24e0 2024-09-15T20:39:38.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8540745e0 con 0x7fb85413c220 2024-09-15T20:39:38.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb83400fed0 con 0x7fb854135b60 2024-09-15T20:39:38.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb8517fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:38.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.262+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:38.468 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.466+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:38.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:38.870+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:39.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:39.674+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 10 monclient(hunting): tick 2024-09-15T20:39:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:39:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 1 monclient(hunting): continuing hunt 2024-09-15T20:39:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:39:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6790/0 conn(0x7fb854135b60 legacy=0x7fb834008eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:41.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.146:6789/0 conn(0x7fb85413c220 legacy=0x7fb83400c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 10 monclient(hunting): picked mon.noname-a con 0x7fb8540a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 10 monclient(hunting): picked mon.noname-b con 0x7fb83400ef00 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 10 monclient(hunting): picked mon.noname-c con 0x7fb85413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8340111d0 con 0x7fb8540a24e0 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb854136650 con 0x7fb85413c220 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 1 -- 172.21.15.78:0/1036603742 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb834006e60 con 0x7fb83400ef00 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb8517fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:41.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.262+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:41.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.466+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:41.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:41.870+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:42.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:42.674+0000 7fb852ffd640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:43.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 886413736 0 0) 0x7fb834006e60 con 0x7fb83400ef00 2024-09-15T20:39:43.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient(hunting): my global_id is 84515 2024-09-15T20:39:43.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb840003530 con 0x7fb83400ef00 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 25733694 0 0) 0x7fb840003530 con 0x7fb83400ef00 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.146:6789/0 conn(0x7fb85413c220 legacy=0x7fb83400c6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb834005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: _finish_hunting 0 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 monclient: found mon.noname-b 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb834008040 con 0x7fb83400ef00 2024-09-15T20:39:43.347 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: _finish_auth 0 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: _check_auth_tickets 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb83c0041a0 con 0x7fb83400ef00 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: dump: 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.348 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 5 monclient: authenticate success, global_id 84515 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb83c004340 con 0x7fb83400ef00 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:43.349 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 1 -- 172.21.15.78:0/1036603742 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb83c0047d0 con 0x7fb83400ef00 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 4 set_mon_vals no callback set 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8527fc640 10 monclient: dump: 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.350 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.351 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 10 monclient: shutdown 2024-09-15T20:39:43.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:43.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 1 -- 172.21.15.78:0/1036603742 >> v1:172.21.15.78:6790/0 conn(0x7fb83400ef00 legacy=0x7fb83400f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 1 -- 172.21.15.78:0/1036603742 shutdown_connections 2024-09-15T20:39:43.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 1 -- 172.21.15.78:0/1036603742 >> 172.21.15.78:0/1036603742 conn(0x7fb8540a63c0 msgr2=0x7fb8540a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:43.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 1 -- 172.21.15.78:0/1036603742 shutdown_connections 2024-09-15T20:39:43.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.342+0000 7fb8592bc640 1 -- 172.21.15.78:0/1036603742 wait complete. 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient: build_initial_monmap 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient: monmap: 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:43.353 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 librados: starting msgr at 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 librados: starting objecter 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 Processor -- start 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 -- start start 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 librados: setting wanted keys 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 librados: calling monclient init 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient: init 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient(hunting): picked mon.noname-a con 0x7fb83400ef00 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient(hunting): picked mon.noname-b con 0x7fb8540a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient(hunting): picked mon.noname-c con 0x7fb85413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85413be00 con 0x7fb83400ef00 2024-09-15T20:39:43.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85409d180 con 0x7fb8540a24e0 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb852ffd640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fb83400ef00 legacy=0x7fb8541cc7b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85413b410 con 0x7fb85413c220 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8592bc640 10 monclient(hunting): authenticate will time out at 1644.234152s 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8537fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb85413c220 0x7fb8541d4af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53184/0 (socket says 172.21.15.78:53184) 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb8537fe640 1 -- 172.21.15.78:0/275752505 learned_addr learned my addr 172.21.15.78:0/275752505 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb84bfff640 10 client.?.objecter ms_handle_connect 0x7fb85413c220 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb84bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb84bfff640 10 client.?.objecter ms_handle_connect 0x7fb8540a24e0 2024-09-15T20:39:43.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.346+0000 7fb84bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:43.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.358+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 78256867 0 0) 0x7fb85413b410 con 0x7fb85413c220 2024-09-15T20:39:43.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.358+0000 7fb84bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.358+0000 7fb84bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.358+0000 7fb84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.358+0000 7fb84bfff640 10 monclient(hunting): my global_id is 84518 2024-09-15T20:39:43.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.358+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb834003500 con 0x7fb85413c220 2024-09-15T20:39:43.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3773417541 0 0) 0x7fb834003500 con 0x7fb85413c220 2024-09-15T20:39:43.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb85413b410 con 0x7fb85413c220 2024-09-15T20:39:43.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb84c001c40 con 0x7fb85413c220 2024-09-15T20:39:43.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:43.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:43.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 10 monclient(hunting): dump: 2024-09-15T20:39:43.364 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.364 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4035205634 0 0) 0x7fb85409d180 con 0x7fb8540a24e0 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 10 monclient(hunting): my global_id is 23566 2024-09-15T20:39:43.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.362+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb834003500 con 0x7fb8540a24e0 2024-09-15T20:39:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2989529153 0 0) 0x7fb85413b410 con 0x7fb85413c220 2024-09-15T20:39:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 >> v1:172.21.15.146:6789/0 conn(0x7fb8540a24e0 legacy=0x7fb854132b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 >> v1:172.21.15.78:6789/0 conn(0x7fb83400ef00 legacy=0x7fb8541cc7b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:39:43.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 monclient: found mon.c 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb85409ee70 con 0x7fb85413c220 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 monclient: _finish_auth 0 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 5 monclient: authenticate success, global_id 84518 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 10 monclient: _renew_subs 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb85409d950 con 0x7fb85413c220 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 10 client.84518.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:43.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 10 monclient: _renew_subs 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb8541333f0 con 0x7fb85413c220 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 1 librados: init done 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb84c003980 con 0x7fb85413c220 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb84c003e10 con 0x7fb85413c220 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb851ffb640 4 set_mon_vals no callback set 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 monclient: dump: 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.370 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.371 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.371 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb84c004030 con 0x7fb85413c220 2024-09-15T20:39:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(342..342 src has 1..342) ==== 5346+0+0 (unknown 1453557862 0 0) 0x7fb84c04b9e0 con 0x7fb85413c220 2024-09-15T20:39:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 client.84518.objecter ms_dispatch 0x7fb854000ba0 osd_map(342..342 src has 1..342) 2024-09-15T20:39:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 3 client.84518.objecter handle_osd_map got epochs [342,342] > 0 2024-09-15T20:39:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 3 client.84518.objecter handle_osd_map decoding full epoch 342 2024-09-15T20:39:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 20 client.84518.objecter dump_active .. 0 homeless 2024-09-15T20:39:43.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb852ffd640 10 monclient: get_auth_request con 0x7fb83403ed30 auth_method 0 2024-09-15T20:39:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb838005090 con 0x7fb85413c220 2024-09-15T20:39:43.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.366+0000 7fb84bfff640 10 client.84518.objecter ms_handle_connect 0x7fb83403ed30 2024-09-15T20:39:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.374+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb84c016dc0 con 0x7fb85413c220 2024-09-15T20:39:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.374+0000 7fb84bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:43.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.374+0000 7fb84bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:39:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb838005380 con 0x7fb85413c220 2024-09-15T20:39:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb84bfff640 1 -- 172.21.15.78:0/275752505 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 3120038641) 0x7fb84c0166c0 con 0x7fb85413c220 2024-09-15T20:39:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb84bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:39:43.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb84bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:43.645 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":34,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 10 librados: watch_flush enter 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 10 librados: watch_flush exit 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 20 client.84518.objecter shutdown clearing up homeless session... 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 10 client.84518.objecter successfully canceled tick 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 >> v1:172.21.15.146:6812/435598642 conn(0x7fb83403ed30 legacy=0x7fb8340411f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 10 monclient: shutdown 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 >> v1:172.21.15.78:6790/0 conn(0x7fb85413c220 legacy=0x7fb8541d4af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 shutdown_connections 2024-09-15T20:39:43.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 >> 172.21.15.78:0/275752505 conn(0x7fb8540a63c0 msgr2=0x7fb8540a46b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:43.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 shutdown_connections 2024-09-15T20:39:43.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 1 -- 172.21.15.78:0/275752505 wait complete. 2024-09-15T20:39:43.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.642+0000 7fb8592bc640 1 librados: shutdown 2024-09-15T20:39:43.663 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:39:43.663 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T20:39:43.663 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:39:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient: get_monmap_and_config 2024-09-15T20:39:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient: build_initial_monmap 2024-09-15T20:39:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient: monmap: 2024-09-15T20:39:43.872 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:43.786153+0000 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:43.786153+0000 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 1 Processor -- start 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 1 -- start start 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient: init 2024-09-15T20:39:43.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient(hunting): picked mon.noname-a con 0x7fa52008d700 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient(hunting): picked mon.noname-b con 0x7fa52009fe50 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient(hunting): picked mon.noname-c con 0x7fa52008dae0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa52009e2b0 con 0x7fa52008d700 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa52009e720 con 0x7fa52008dae0 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa52009f9a0 con 0x7fa52009fe50 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa528507640 10 monclient(hunting): authenticate will time out at 1644.670691s 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa5262a3640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fa52008d700 legacy=0x7fa52009d490 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa525aa2640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa52009fe50 0x7fa52009dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53188/0 (socket says 172.21.15.78:53188) 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa525aa2640 1 -- 172.21.15.78:0/1672976023 learned_addr learned my addr 172.21.15.78:0/1672976023 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:43.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3615536253 0 0) 0x7fa52009f9a0 con 0x7fa52009fe50 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa5252a1640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa5252a1640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa5252a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa5252a1640 10 monclient(hunting): my global_id is 84521 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.782+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa508003530 con 0x7fa52009fe50 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.866+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3941080745 0 0) 0x7fa52009e720 con 0x7fa52008dae0 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient(hunting): my global_id is 23569 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa52009f9a0 con 0x7fa52008dae0 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 301337149 0 0) 0x7fa508003530 con 0x7fa52009fe50 2024-09-15T20:39:43.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 >> v1:172.21.15.146:6789/0 conn(0x7fa52008dae0 legacy=0x7fa52014d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 >> v1:172.21.15.78:6789/0 conn(0x7fa52008d700 legacy=0x7fa52009d490 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: _finish_hunting 0 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 monclient: found mon.noname-b 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5200a38a0 con 0x7fa52009fe50 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: _finish_auth 0 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: _check_auth_tickets 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa514001a20 con 0x7fa52009fe50 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:43.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: dump: 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 5 monclient: authenticate success, global_id 84521 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:43.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa514001bc0 con 0x7fa52009fe50 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 1 -- 172.21.15.78:0/1672976023 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa514002010 con 0x7fa52009fe50 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 4 set_mon_vals no callback set 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa5252a1640 10 monclient: dump: 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.878 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.879 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.879 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.879 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.879 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient: shutdown 2024-09-15T20:39:43.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:43.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 -- 172.21.15.78:0/1672976023 >> v1:172.21.15.78:6790/0 conn(0x7fa52009fe50 legacy=0x7fa52009dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 -- 172.21.15.78:0/1672976023 shutdown_connections 2024-09-15T20:39:43.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 -- 172.21.15.78:0/1672976023 >> 172.21.15.78:0/1672976023 conn(0x7fa5200a2a90 msgr2=0x7fa5200a2e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:43.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 -- 172.21.15.78:0/1672976023 shutdown_connections 2024-09-15T20:39:43.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 -- 172.21.15.78:0/1672976023 wait complete. 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient: build_initial_monmap 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient: monmap: 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 librados: starting msgr at 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 librados: starting objecter 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 Processor -- start 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 -- start start 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 librados: setting wanted keys 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 1 librados: calling monclient init 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient: init 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient(hunting): picked mon.noname-b con 0x7fa52008d700 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.870+0000 7fa528507640 10 monclient(hunting): picked mon.noname-c con 0x7fa52008dae0 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 monclient(hunting): picked mon.noname-a con 0x7fa52009fe50 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa520084c50 con 0x7fa52009fe50 2024-09-15T20:39:43.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa52009d180 con 0x7fa52008d700 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5200828f0 con 0x7fa52008dae0 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 monclient(hunting): authenticate will time out at 1644.760070s 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa526aa4640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fa52009fe50 legacy=0x7fa5201e1c20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa525aa2640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa52008dae0 0x7fa5200817a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53194/0 (socket says 172.21.15.78:53194) 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa525aa2640 1 -- 172.21.15.78:0/703144346 learned_addr learned my addr 172.21.15.78:0/703144346 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 client.?.objecter ms_handle_connect 0x7fa52008dae0 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4288680900 0 0) 0x7fa5200828f0 con 0x7fa52008dae0 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): my global_id is 84524 2024-09-15T20:39:43.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa50c004ec0 con 0x7fa52008dae0 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 client.?.objecter ms_handle_connect 0x7fa52008d700 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2457357922 0 0) 0x7fa52009d180 con 0x7fa52008d700 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): my global_id is 23572 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5200828f0 con 0x7fa52008d700 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 602102417 0 0) 0x7fa50c004ec0 con 0x7fa52008dae0 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa52009d180 con 0x7fa52008dae0 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa514001f10 con 0x7fa52008dae0 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:43.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): dump: 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3203657066 0 0) 0x7fa52009d180 con 0x7fa52008dae0 2024-09-15T20:39:43.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 >> v1:172.21.15.146:6789/0 conn(0x7fa52008d700 legacy=0x7fa520081090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 >> v1:172.21.15.78:6789/0 conn(0x7fa52009fe50 legacy=0x7fa5201e1c20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: _finish_hunting 0 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 monclient: found mon.c 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5200843b0 con 0x7fa52008dae0 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: _finish_auth 0 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: _check_auth_tickets 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 5 monclient: authenticate success, global_id 84524 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 monclient: _renew_subs 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa520083450 con 0x7fa52008dae0 2024-09-15T20:39:43.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 client.84524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 monclient: _renew_subs 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa5200840a0 con 0x7fa52008dae0 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 1 librados: init done 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa514004f80 con 0x7fa52008dae0 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa5140052d0 con 0x7fa52008dae0 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: dump: 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.891 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa524aa0640 4 set_mon_vals no callback set 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa5200008e0 con 0x7fa520081f10 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fa514011340 con 0x7fa52008dae0 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 client.84524.objecter ms_handle_connect 0x7fa520081f10 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 client.84524.objecter resend_mon_ops 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(343..343 src has 1..343) ==== 5375+0+0 (unknown 1919621293 0 0) 0x7fa51404bb90 con 0x7fa52008dae0 2024-09-15T20:39:43.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 client.84524.objecter ms_dispatch 0x7fa520000ba0 osd_map(343..343 src has 1..343) 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 3 client.84524.objecter handle_osd_map got epochs [343,343] > 0 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 3 client.84524.objecter handle_osd_map decoding full epoch 343 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 20 client.84524.objecter dump_active .. 0 homeless 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1170286365 0 0) 0x7fa5200008e0 con 0x7fa520081f10 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient(hunting): my global_id is 84527 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa52009d180 con 0x7fa520081f10 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa526aa4640 10 monclient: get_auth_request con 0x7fa50c03e000 auth_method 0 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3571304715 0 0) 0x7fa52009d180 con 0x7fa520081f10 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5200008e0 con 0x7fa520081f10 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa5180040f0 con 0x7fa520081f10 2024-09-15T20:39:43.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 monclient: dump: 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3527200521 0 0) 0x7fa5200008e0 con 0x7fa520081f10 2024-09-15T20:39:43.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa520069830 con 0x7fa520081f10 2024-09-15T20:39:43.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.874+0000 7fa51effd640 10 client.84524.objecter ms_handle_connect 0x7fa50c03e000 2024-09-15T20:39:43.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.878+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fa518006430 con 0x7fa520081f10 2024-09-15T20:39:43.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.878+0000 7fa51effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:43.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.878+0000 7fa51effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:43.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.878+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 >> v1:172.21.15.78:6790/0 conn(0x7fa520081f10 legacy=0x7fa52007cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.878+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 >> v1:172.21.15.78:6790/0 conn(0x7fa520081f10 legacy=0x7fa52007cfa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:43.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa528507640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fa520084890 con 0x7fa52002e460 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 client.84524.objecter ms_handle_connect 0x7fa52002e460 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 client.84524.objecter resend_mon_ops 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4089818165 0 0) 0x7fa5200828f0 con 0x7fa52002e460 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient(hunting): my global_id is 84530 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa5200008e0 con 0x7fa52002e460 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 15796390 0 0) 0x7fa5200008e0 con 0x7fa52002e460 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa5200828f0 con 0x7fa52002e460 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa5180047a0 con 0x7fa52002e460 2024-09-15T20:39:43.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient: dump: 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1139641782 0 0) 0x7fa5200828f0 con 0x7fa52002e460 2024-09-15T20:39:43.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa52002e870 con 0x7fa52002e460 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4127 (unknown 0 0 2170197648) 0x7fa518005cd0 con 0x7fa52002e460 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 >> v1:172.21.15.78:6790/0 conn(0x7fa52002e460 legacy=0x7fa52007d6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa51effd640 1 -- 172.21.15.78:0/703144346 >> v1:172.21.15.78:6790/0 conn(0x7fa52002e460 legacy=0x7fa52007d6b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:43.899 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 34, 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 255569, 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:43.900 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:43.901 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:39:43.902 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:43.903 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:43.904 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:43.905 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:39:43.906 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:43.907 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:39:43.908 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:43.909 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:39:43.910 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:39:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa528507640 10 librados: watch_flush enter 2024-09-15T20:39:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa528507640 10 librados: watch_flush exit 2024-09-15T20:39:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa528507640 20 client.84524.objecter shutdown clearing up homeless session... 2024-09-15T20:39:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.882+0000 7fa528507640 10 client.84524.objecter successfully canceled tick 2024-09-15T20:39:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 >> v1:172.21.15.146:6812/435598642 conn(0x7fa50c03e000 legacy=0x7fa50c0404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 10 monclient: shutdown 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 >> v1:172.21.15.78:6790/0 conn(0x7fa52008dae0 legacy=0x7fa5200817a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa526aa4640 1 -- 172.21.15.78:0/703144346 reap_dead start 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa526aa4640 1 -- 172.21.15.78:0/703144346 reap_dead start 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 shutdown_connections 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 >> 172.21.15.78:0/703144346 conn(0x7fa5200a2a90 msgr2=0x7fa5200a4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 shutdown_connections 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 1 -- 172.21.15.78:0/703144346 wait complete. 2024-09-15T20:39:43.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:43.886+0000 7fa528507640 1 librados: shutdown 2024-09-15T20:39:43.914 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:39:43.914 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T20:39:43.914 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:39:44.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient: get_monmap_and_config 2024-09-15T20:39:44.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient: build_initial_monmap 2024-09-15T20:39:44.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient: monmap: 2024-09-15T20:39:44.032 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:39:44.033888+0000 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:39:44.033888+0000 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:39:44.033 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:44.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 Processor -- start 2024-09-15T20:39:44.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- start start 2024-09-15T20:39:44.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient: init 2024-09-15T20:39:44.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient(hunting): picked mon.noname-b con 0x7fd25c135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient(hunting): picked mon.noname-a con 0x7fd25c0a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient(hunting): picked mon.noname-c con 0x7fd25c13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c0745e0 con 0x7fd25c0a24e0 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c06ecf0 con 0x7fd25c13c220 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c06efd0 con 0x7fd25c135b60 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient(hunting): authenticate will time out at 1644.918398s 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25b7fe640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fd25c0a24e0 legacy=0x7fd25c13ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25bfff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd25c135b60 0x7fd25c135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53224/0 (socket says 172.21.15.78:53224) 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25bfff640 1 -- 172.21.15.78:0/3605449045 learned_addr learned my addr 172.21.15.78:0/3605449045 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:44.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2156304362 0 0) 0x7fd25c06efd0 con 0x7fd25c135b60 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient(hunting): _negotiate 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient(hunting): my global_id is 84533 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd244003530 con 0x7fd25c135b60 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3613361790 0 0) 0x7fd244003530 con 0x7fd25c135b60 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 >> v1:172.21.15.146:6789/0 conn(0x7fd25c13c220 legacy=0x7fd25c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 >> v1:172.21.15.78:6789/0 conn(0x7fd25c0a24e0 legacy=0x7fd25c13ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: _finish_hunting 0 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 monclient: found mon.noname-b 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c136650 con 0x7fd25c135b60 2024-09-15T20:39:44.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: _finish_auth 0 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: _check_auth_tickets 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd24c0016a0 con 0x7fd25c135b60 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 5 monclient: authenticate success, global_id 84533 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: dump: 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:44.043 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd24c0018c0 con 0x7fd25c135b60 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 1 -- 172.21.15.78:0/3605449045 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd24c001cd0 con 0x7fd25c135b60 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient: get_monmap_and_config success 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 4 set_mon_vals no callback set 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:44.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd25affd640 10 monclient: dump: 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:44.045 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 10 monclient: shutdown 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- 172.21.15.78:0/3605449045 >> v1:172.21.15.78:6790/0 conn(0x7fd25c135b60 legacy=0x7fd25c135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- 172.21.15.78:0/3605449045 shutdown_connections 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- 172.21.15.78:0/3605449045 >> 172.21.15.78:0/3605449045 conn(0x7fd25c0a63c0 msgr2=0x7fd25c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- 172.21.15.78:0/3605449045 shutdown_connections 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.030+0000 7fd262913640 1 -- 172.21.15.78:0/3605449045 wait complete. 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: build_initial_monmap 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: monmap: 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:39:44.046 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 librados: starting msgr at 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 librados: starting objecter 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 Processor -- start 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 -- start start 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 librados: setting wanted keys 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 librados: calling monclient init 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: init 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: _reopen_session rank -1 2024-09-15T20:39:44.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient(hunting): picked mon.noname-b con 0x7fd25c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient(hunting): picked mon.noname-c con 0x7fd25c135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient(hunting): picked mon.noname-a con 0x7fd25c13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c1385d0 con 0x7fd25c13c220 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c13f9c0 con 0x7fd25c0a24e0 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd25c09d180 con 0x7fd25c135b60 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient(hunting): _renew_subs 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient(hunting): authenticate will time out at 1644.921284s 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd260eb0640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fd25c13c220 legacy=0x7fd25c072c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd25b7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd25c135b60 0x7fd25c072550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53230/0 (socket says 172.21.15.78:53230) 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd25b7fe640 1 -- 172.21.15.78:0/2085034876 learned_addr learned my addr 172.21.15.78:0/2085034876 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:39:44.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 client.?.objecter ms_handle_connect 0x7fd25c135b60 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1133612127 0 0) 0x7fd25c09d180 con 0x7fd25c135b60 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): my global_id is 84536 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd248004ef0 con 0x7fd25c135b60 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 client.?.objecter ms_handle_connect 0x7fd25c0a24e0 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4019893504 0 0) 0x7fd248004ef0 con 0x7fd25c135b60 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd25c09d180 con 0x7fd25c135b60 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd2500041e0 con 0x7fd25c135b60 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:39:44.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): dump: 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 891163630 0 0) 0x7fd25c09d180 con 0x7fd25c135b60 2024-09-15T20:39:44.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 >> v1:172.21.15.146:6789/0 conn(0x7fd25c0a24e0 legacy=0x7fd25c071e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 >> v1:172.21.15.78:6789/0 conn(0x7fd25c13c220 legacy=0x7fd25c072c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 monclient: found mon.c 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd25c1382f0 con 0x7fd25c135b60 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient: _finish_auth 0 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:39:44.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 5 monclient: authenticate success, global_id 84536 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: _renew_subs 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd25c13b6f0 con 0x7fd25c135b60 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 client.84536.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: _renew_subs 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd25c1d3df0 con 0x7fd25c135b60 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 librados: init done 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd250004400 con 0x7fd25c135b60 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd250004810 con 0x7fd25c135b60 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:39:44.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient: dump: 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd25a7fc640 4 set_mon_vals no callback set 2024-09-15T20:39:44.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd23c0065d0 con 0x7fd23c002870 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd250004a30 con 0x7fd25c135b60 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 client.84536.objecter ms_handle_connect 0x7fd23c002870 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 client.84536.objecter resend_mon_ops 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(343..343 src has 1..343) ==== 5375+0+0 (unknown 1919621293 0 0) 0x7fd25004b8d0 con 0x7fd25c135b60 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 client.84536.objecter ms_dispatch 0x7fd25c000ba0 osd_map(343..343 src has 1..343) 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 3 client.84536.objecter handle_osd_map got epochs [343,343] > 0 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 3 client.84536.objecter handle_osd_map decoding full epoch 343 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 20 client.84536.objecter dump_active .. 0 homeless 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3032668723 0 0) 0x7fd23c0065d0 con 0x7fd23c002870 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:39:44.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 10 monclient(hunting): my global_id is 23581 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd25c09d180 con 0x7fd23c002870 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.034+0000 7fd260eb0640 10 monclient: get_auth_request con 0x7fd24803dec0 auth_method 0 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3949133353 0 0) 0x7fd25c09d180 con 0x7fd23c002870 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd23c0065d0 con 0x7fd23c002870 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd24c001660 con 0x7fd23c002870 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 10 monclient: dump: 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.055 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 10 client.84536.objecter ms_handle_connect 0x7fd24803dec0 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2871741207 0 0) 0x7fd23c0065d0 con 0x7fd23c002870 2024-09-15T20:39:44.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd23c00a170 con 0x7fd23c002870 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fd24c0018e0 con 0x7fd23c002870 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 >> v1:172.21.15.146:6789/0 conn(0x7fd23c002870 legacy=0x7fd23c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.038+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 >> v1:172.21.15.146:6789/0 conn(0x7fd23c002870 legacy=0x7fd23c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.042+0000 7fd262913640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.042+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fd23c009e80 con 0x7fd23c009140 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 client.84536.objecter ms_handle_connect 0x7fd23c009140 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 client.84536.objecter resend_mon_ops 2024-09-15T20:39:44.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2727670707 0 0) 0x7fd23c009e80 con 0x7fd23c009140 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient(hunting): my global_id is 23584 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd23c0065d0 con 0x7fd23c009140 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3277431448 0 0) 0x7fd23c0065d0 con 0x7fd23c009140 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd23c009e80 con 0x7fd23c009140 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd24c004af0 con 0x7fd23c009140 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient: dump: 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:39:44.059 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3190615133 0 0) 0x7fd23c009e80 con 0x7fd23c009140 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd23c00c420 con 0x7fd23c009140 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 2230942992) 0x7fd24c004a80 con 0x7fd23c009140 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:39:44.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 >> v1:172.21.15.146:6789/0 conn(0x7fd23c009140 legacy=0x7fd23c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd258ff9640 1 -- 172.21.15.78:0/2085034876 >> v1:172.21.15.146:6789/0 conn(0x7fd23c009140 legacy=0x7fd23c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 34, 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 496554, 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:39:44.061 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:44.062 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:44.063 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:44.064 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:39:44.065 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:39:44.066 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:44.067 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:44.068 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.069 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:39:44.070 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:44.071 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:39:44.072 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:39:44.073 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:39:44.073 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:39:44.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 10 librados: watch_flush enter 2024-09-15T20:39:44.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 10 librados: watch_flush exit 2024-09-15T20:39:44.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 20 client.84536.objecter shutdown clearing up homeless session... 2024-09-15T20:39:44.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 10 client.84536.objecter successfully canceled tick 2024-09-15T20:39:44.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 >> v1:172.21.15.146:6812/435598642 conn(0x7fd24803dec0 legacy=0x7fd248040380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:44.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 10 monclient: shutdown 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 >> v1:172.21.15.78:6790/0 conn(0x7fd25c135b60 legacy=0x7fd25c072550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd260eb0640 1 -- 172.21.15.78:0/2085034876 reap_dead start 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd260eb0640 1 -- 172.21.15.78:0/2085034876 reap_dead start 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 shutdown_connections 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 >> 172.21.15.78:0/2085034876 conn(0x7fd25c0a63c0 msgr2=0x7fd25c12e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 shutdown_connections 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 1 -- 172.21.15.78:0/2085034876 wait complete. 2024-09-15T20:39:44.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:39:44.046+0000 7fd262913640 1 librados: shutdown 2024-09-15T20:39:44.075 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:39:44.076 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:41:14.076 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T20:41:14.077 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T20:41:14.077 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T20:41:14.081 INFO:tasks.ceph.mon.a:Started 2024-09-15T20:41:14.081 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:41:14.081 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:41:14.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: get_monmap_and_config 2024-09-15T20:41:14.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: build_initial_monmap 2024-09-15T20:41:14.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: monmap: 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:14.208982+0000 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:14.208982+0000 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:14.208 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 Processor -- start 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- start start 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: init 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient(hunting): picked mon.noname-a con 0x7fabc8137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient(hunting): picked mon.noname-c con 0x7fabc80a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient(hunting): picked mon.noname-b con 0x7fabc813e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc80a6420 con 0x7fabc8137ce0 2024-09-15T20:41:14.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc80a6700 con 0x7fabc80a9610 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc80a2110 con 0x7fabc813e5f0 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient(hunting): authenticate will time out at 1735.093280s 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcdc21640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fabc813e5f0 0x7fabc81409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35282/0 (socket says 172.21.15.78:35282) 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcdc21640 1 -- 172.21.15.78:0/1559752297 learned_addr learned my addr 172.21.15.78:0/1559752297 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcd420640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fabc8137ce0 0x7fabc81380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42218/0 (socket says 172.21.15.78:42218) 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4247178366 0 0) 0x7fabc80a2110 con 0x7fabc813e5f0 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient(hunting): my global_id is 84539 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabb4003530 con 0x7fabc813e5f0 2024-09-15T20:41:14.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2908093269 0 0) 0x7fabb4003530 con 0x7fabc813e5f0 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 >> v1:172.21.15.146:6789/0 conn(0x7fabc80a9610 legacy=0x7fabc813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 >> v1:172.21.15.78:6789/0 conn(0x7fabc8137ce0 legacy=0x7fabc81380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 monclient: found mon.noname-b 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc80a4bc0 con 0x7fabc813e5f0 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fabc0001a20 con 0x7fabc813e5f0 2024-09-15T20:41:14.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:41:14.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: dump: 2024-09-15T20:41:14.219 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.219 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.219 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 5 monclient: authenticate success, global_id 84539 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fabc0001c40 con 0x7fabc813e5f0 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:14.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 1 -- 172.21.15.78:0/1559752297 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fabc0002050 con 0x7fabc813e5f0 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 4 set_mon_vals no callback set 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabbffff640 10 monclient: dump: 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.221 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: shutdown 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- 172.21.15.78:0/1559752297 >> v1:172.21.15.78:6790/0 conn(0x7fabc813e5f0 legacy=0x7fabc81409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- 172.21.15.78:0/1559752297 shutdown_connections 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- 172.21.15.78:0/1559752297 >> 172.21.15.78:0/1559752297 conn(0x7fabc80aa6c0 msgr2=0x7fabc80aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- 172.21.15.78:0/1559752297 shutdown_connections 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- 172.21.15.78:0/1559752297 wait complete. 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: build_initial_monmap 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: monmap: 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:14.222 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 librados: starting msgr at 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 librados: starting objecter 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 Processor -- start 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 -- start start 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 librados: setting wanted keys 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 1 librados: calling monclient init 2024-09-15T20:41:14.223 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.205+0000 7fabcf684640 10 monclient: init 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient(hunting): picked mon.noname-b con 0x7fabc80a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient(hunting): picked mon.noname-a con 0x7fabc8137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient(hunting): picked mon.noname-c con 0x7fabc813e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc813c240 con 0x7fabc8137ce0 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc8139cd0 con 0x7fabc80a9610 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fabc8074230 con 0x7fabc813e5f0 2024-09-15T20:41:14.224 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient(hunting): authenticate will time out at 1735.096191s 2024-09-15T20:41:14.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcdc21640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fabc813e5f0 0x7fabc81d4fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35296/0 (socket says 172.21.15.78:35296) 2024-09-15T20:41:14.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabccc1f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fabc8137ce0 0x7fabc81395c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42230/0 (socket says 172.21.15.78:42230) 2024-09-15T20:41:14.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcdc21640 1 -- 172.21.15.78:0/3500164075 learned_addr learned my addr 172.21.15.78:0/3500164075 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:14.225 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 client.?.objecter ms_handle_connect 0x7fabc8137ce0 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 client.?.objecter ms_handle_connect 0x7fabc813e5f0 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3361512628 0 0) 0x7fabc8074230 con 0x7fabc813e5f0 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient(hunting): my global_id is 84542 2024-09-15T20:41:14.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fabac004ec0 con 0x7fabc813e5f0 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 client.?.objecter ms_handle_connect 0x7fabc80a9610 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2516071584 0 0) 0x7fabac004ec0 con 0x7fabc813e5f0 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fabc8074230 con 0x7fabc813e5f0 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fabc0001f90 con 0x7fabc813e5f0 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient(hunting): dump: 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.227 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 458951256 0 0) 0x7fabc8074230 con 0x7fabc813e5f0 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 >> v1:172.21.15.146:6789/0 conn(0x7fabc80a9610 legacy=0x7fabc8138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 >> v1:172.21.15.78:6789/0 conn(0x7fabc8137ce0 legacy=0x7fabc81395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 monclient: found mon.c 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabc813aa40 con 0x7fabc813e5f0 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 5 monclient: authenticate success, global_id 84542 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient: _renew_subs 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fabc80a0190 con 0x7fabc813e5f0 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 client.84542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient: _renew_subs 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fabc81d6700 con 0x7fabc813e5f0 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 1 librados: init done 2024-09-15T20:41:14.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fabc0001530 con 0x7fabc813e5f0 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fabc0005120 con 0x7fabc813e5f0 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbf7fe640 4 set_mon_vals no callback set 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 monclient: dump: 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.230 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faba8005090 con 0x7fabc813e5f0 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fabc0011190 con 0x7fabc813e5f0 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabcd420640 10 monclient: get_auth_request con 0x7fabac03dfc0 auth_method 0 2024-09-15T20:41:14.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.209+0000 7fabbdffb640 10 client.84542.objecter ms_handle_connect 0x7fabac03dfc0 2024-09-15T20:41:14.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.233+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(379..379 src has 223..379) ==== 5346+0+0 (unknown 1175008561 0 0) 0x7fabc004a310 con 0x7fabc813e5f0 2024-09-15T20:41:14.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.233+0000 7fabbdffb640 10 client.84542.objecter ms_dispatch 0x7fabc8000ba0 osd_map(379..379 src has 223..379) 2024-09-15T20:41:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.233+0000 7fabbdffb640 3 client.84542.objecter handle_osd_map got epochs [379,379] > 0 2024-09-15T20:41:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.233+0000 7fabbdffb640 3 client.84542.objecter handle_osd_map decoding full epoch 379 2024-09-15T20:41:14.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.233+0000 7fabbdffb640 20 client.84542.objecter dump_active .. 0 homeless 2024-09-15T20:41:14.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.237+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fabc0016f30 con 0x7fabc813e5f0 2024-09-15T20:41:14.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.237+0000 7fabbdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.237+0000 7fabbdffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:14.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:41:14.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:14.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7faba8005380 con 0x7fabc813e5f0 2024-09-15T20:41:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabbdffb640 1 -- 172.21.15.78:0/3500164075 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3347287723) 0x7fabc0016830 con 0x7fabc813e5f0 2024-09-15T20:41:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabbdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:14.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabbdffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:14.473 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:41:14.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 10 librados: watch_flush enter 2024-09-15T20:41:14.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 10 librados: watch_flush exit 2024-09-15T20:41:14.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 20 client.84542.objecter shutdown clearing up homeless session... 2024-09-15T20:41:14.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 10 client.84542.objecter successfully canceled tick 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 >> v1:172.21.15.146:6812/435598642 conn(0x7fabac03dfc0 legacy=0x7fabac040480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 10 monclient: shutdown 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.469+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 >> v1:172.21.15.78:6790/0 conn(0x7fabc813e5f0 legacy=0x7fabc81d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.473+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 shutdown_connections 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.473+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 >> 172.21.15.78:0/3500164075 conn(0x7fabc80aa6c0 msgr2=0x7fabc8073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.473+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 shutdown_connections 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.473+0000 7fabcf684640 1 -- 172.21.15.78:0/3500164075 wait complete. 2024-09-15T20:41:14.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.473+0000 7fabcf684640 1 librados: shutdown 2024-09-15T20:41:14.491 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:41:14.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:41:14.491 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:41:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.605+0000 7f891c68a640 10 monclient: get_monmap_and_config 2024-09-15T20:41:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.605+0000 7f891c68a640 10 monclient: build_initial_monmap 2024-09-15T20:41:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.605+0000 7f891c68a640 10 monclient: monmap: 2024-09-15T20:41:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:14.611 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.612 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:14.612827+0000 2024-09-15T20:41:14.612 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:14.612827+0000 2024-09-15T20:41:14.612 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:14.612 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.612 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:14.612 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:14.612 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:14.612 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 Processor -- start 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- start start 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient: init 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient(hunting): picked mon.noname-b con 0x7f8914135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient(hunting): picked mon.noname-a con 0x7f89140a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient(hunting): picked mon.noname-c con 0x7f891413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89140745e0 con 0x7f89140a24e0 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f891406ecf0 con 0x7f891413c220 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f891406efd0 con 0x7f8914135b60 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient(hunting): authenticate will time out at 1735.497410s 2024-09-15T20:41:14.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891a426640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f8914135b60 0x7f8914135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35302/0 (socket says 172.21.15.78:35302) 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891a426640 1 -- 172.21.15.78:0/1151585434 learned_addr learned my addr 172.21.15.78:0/1151585434 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3919341219 0 0) 0x7f891406efd0 con 0x7f8914135b60 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient(hunting): my global_id is 84545 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88fc003530 con 0x7f8914135b60 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1878878861 0 0) 0x7f89140745e0 con 0x7f89140a24e0 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient(hunting): my global_id is 114099 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f891406efd0 con 0x7f89140a24e0 2024-09-15T20:41:14.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3188732161 0 0) 0x7f88fc003530 con 0x7f8914135b60 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 >> v1:172.21.15.146:6789/0 conn(0x7f891413c220 legacy=0x7f891413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 >> v1:172.21.15.78:6789/0 conn(0x7f89140a24e0 legacy=0x7f891413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 monclient: found mon.noname-b 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8914136650 con 0x7f8914135b60 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 5 monclient: authenticate success, global_id 84545 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:14.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f890c0016a0 con 0x7f8914135b60 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: dump: 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f890c001840 con 0x7f8914135b60 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 1 -- 172.21.15.78:0/1151585434 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f890c001c10 con 0x7f8914135b60 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f8919424640 10 monclient: dump: 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.619 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 4 set_mon_vals no callback set 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 10 monclient: shutdown 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- 172.21.15.78:0/1151585434 >> v1:172.21.15.78:6790/0 conn(0x7f8914135b60 legacy=0x7f8914135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- 172.21.15.78:0/1151585434 shutdown_connections 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- 172.21.15.78:0/1151585434 >> 172.21.15.78:0/1151585434 conn(0x7f89140a63c0 msgr2=0x7f89140a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- 172.21.15.78:0/1151585434 shutdown_connections 2024-09-15T20:41:14.620 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.609+0000 7f891c68a640 1 -- 172.21.15.78:0/1151585434 wait complete. 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: build_initial_monmap 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: monmap: 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.626 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 librados: starting msgr at 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 librados: starting objecter 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 Processor -- start 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 -- start start 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 librados: setting wanted keys 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 librados: calling monclient init 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: init 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient(hunting): picked mon.noname-b con 0x7f89140a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient(hunting): picked mon.noname-c con 0x7f8914135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient(hunting): picked mon.noname-a con 0x7f891413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8914072860 con 0x7f891413c220 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f891409d180 con 0x7f89140a24e0 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89140a37b0 con 0x7f8914135b60 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient(hunting): authenticate will time out at 1735.501266s 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8919c25640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f8914135b60 0x7f8914072150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35304/0 (socket says 172.21.15.78:35304) 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8919c25640 1 -- 172.21.15.78:0/722894437 learned_addr learned my addr 172.21.15.78:0/722894437 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.?.objecter ms_handle_connect 0x7f8914135b60 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.?.objecter ms_handle_connect 0x7f891413c220 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3979211745 0 0) 0x7f89140a37b0 con 0x7f8914135b60 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): my global_id is 84548 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8900004ec0 con 0x7f8914135b60 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1754427506 0 0) 0x7f8914072860 con 0x7f891413c220 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): my global_id is 114102 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89140a37b0 con 0x7f891413c220 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.?.objecter ms_handle_connect 0x7f89140a24e0 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3944192364 0 0) 0x7f8900004ec0 con 0x7f8914135b60 2024-09-15T20:41:14.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8914072860 con 0x7f8914135b60 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f89080013c0 con 0x7f8914135b60 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): dump: 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.630 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4268658864 0 0) 0x7f89140a37b0 con 0x7f891413c220 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8900004ec0 con 0x7f891413c220 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8904001b80 con 0x7f891413c220 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): dump: 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.631 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3147592172 0 0) 0x7f891409d180 con 0x7f89140a24e0 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): my global_id is 23596 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f89140a37b0 con 0x7f89140a24e0 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3520807085 0 0) 0x7f8914072860 con 0x7f8914135b60 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 >> v1:172.21.15.146:6789/0 conn(0x7f89140a24e0 legacy=0x7f8914071a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 >> v1:172.21.15.78:6789/0 conn(0x7f891413c220 legacy=0x7f89141d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 monclient: found mon.c 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f891409d590 con 0x7f8914135b60 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 5 monclient: authenticate success, global_id 84548 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: _renew_subs 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f891409e620 con 0x7f8914135b60 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 client.84548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: _renew_subs 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8914135480 con 0x7f8914135b60 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 librados: init done 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f89080038b0 con 0x7f8914135b60 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8908003aa0 con 0x7f8914135b60 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8918c23640 4 set_mon_vals no callback set 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: dump: 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.634 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f88f40065d0 con 0x7f88f4002870 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.84548.objecter ms_handle_connect 0x7f88f4002870 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.84548.objecter resend_mon_ops 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1639718088 0 0) 0x7f88f40065d0 con 0x7f88f4002870 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient(hunting): my global_id is 114105 2024-09-15T20:41:14.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8914072860 con 0x7f88f4002870 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f8908003cc0 con 0x7f8914135b60 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(379..379 src has 223..379) ==== 5346+0+0 (unknown 1175008561 0 0) 0x7f890804b740 con 0x7f8914135b60 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 client.84548.objecter ms_dispatch 0x7f8914000ba0 osd_map(379..379 src has 223..379) 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 3 client.84548.objecter handle_osd_map got epochs [379,379] > 0 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 3 client.84548.objecter handle_osd_map decoding full epoch 379 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 20 client.84548.objecter dump_active .. 0 homeless 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3248365171 0 0) 0x7f8914072860 con 0x7f88f4002870 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88f40065d0 con 0x7f88f4002870 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f890c001510 con 0x7f88f4002870 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 10 monclient: dump: 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.636 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2301468308 0 0) 0x7f88f40065d0 con 0x7f88f4002870 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88f400a170 con 0x7f88f4002870 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.613+0000 7f891ac27640 10 monclient: get_auth_request con 0x7f890003e260 auth_method 0 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.617+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f890c0015e0 con 0x7f88f4002870 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.617+0000 7f8912ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.617+0000 7f8912ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:14.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.617+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 >> v1:172.21.15.78:6789/0 conn(0x7f88f4002870 legacy=0x7f88f4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.617+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 >> v1:172.21.15.78:6789/0 conn(0x7f88f4002870 legacy=0x7f88f4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.617+0000 7f8912ffd640 10 client.84548.objecter ms_handle_connect 0x7f890003e260 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f88f4009e80 con 0x7f88f4009140 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 client.84548.objecter ms_handle_connect 0x7f88f4009140 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 client.84548.objecter resend_mon_ops 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2328789816 0 0) 0x7f88f4009e80 con 0x7f88f4009140 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient(hunting): my global_id is 114108 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f88f40065d0 con 0x7f88f4009140 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2518112697 0 0) 0x7f88f40065d0 con 0x7f88f4009140 2024-09-15T20:41:14.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f88f4009e80 con 0x7f88f4009140 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f890c0047f0 con 0x7f88f4009140 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient: dump: 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 73926303 0 0) 0x7f88f4009e80 con 0x7f88f4009140 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88f400c400 con 0x7f88f4009140 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3970 (unknown 0 0 1378164979) 0x7f890c004a10 con 0x7f88f4009140 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 >> v1:172.21.15.78:6789/0 conn(0x7f88f4009140 legacy=0x7f88f40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f8912ffd640 1 -- 172.21.15.78:0/722894437 >> v1:172.21.15.78:6789/0 conn(0x7f88f4009140 legacy=0x7f88f40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:14.640 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 38, 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 463, 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.641 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.642 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.643 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:41:14.644 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.645 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.646 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.647 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:14.648 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.649 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.650 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.651 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:41:14.651 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 10 librados: watch_flush enter 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 10 librados: watch_flush exit 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 20 client.84548.objecter shutdown clearing up homeless session... 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 10 client.84548.objecter successfully canceled tick 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 >> v1:172.21.15.146:6812/435598642 conn(0x7f890003e260 legacy=0x7f8900040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 10 monclient: shutdown 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 >> v1:172.21.15.78:6790/0 conn(0x7f8914135b60 legacy=0x7f8914072150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891ac27640 1 -- 172.21.15.78:0/722894437 reap_dead start 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891ac27640 1 -- 172.21.15.78:0/722894437 reap_dead start 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 shutdown_connections 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 >> 172.21.15.78:0/722894437 conn(0x7f89140a63c0 msgr2=0x7f891412e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:14.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 shutdown_connections 2024-09-15T20:41:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 1 -- 172.21.15.78:0/722894437 wait complete. 2024-09-15T20:41:14.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.621+0000 7f891c68a640 1 librados: shutdown 2024-09-15T20:41:14.654 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:41:14.654 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:41:14.654 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: get_monmap_and_config 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: build_initial_monmap 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: monmap: 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:14.773435+0000 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:14.773435+0000 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.772 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:14.773 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:14.773 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:14.773 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 Processor -- start 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- start start 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: init 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient(hunting): picked mon.noname-a con 0x7f50f4137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient(hunting): picked mon.noname-c con 0x7f50f40a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient(hunting): picked mon.noname-b con 0x7f50f413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f40a6420 con 0x7f50f4137ce0 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f40a6700 con 0x7f50f40a9610 2024-09-15T20:41:14.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f40a2110 con 0x7f50f413e5f0 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient(hunting): authenticate will time out at 1735.657963s 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50f940a640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f50f4137ce0 0x7f50f41380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42280/0 (socket says 172.21.15.78:42280) 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50f940a640 1 -- 172.21.15.78:0/4201547052 learned_addr learned my addr 172.21.15.78:0/4201547052 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 25750285 0 0) 0x7f50f40a6420 con 0x7f50f4137ce0 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient(hunting): my global_id is 114111 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50e0003530 con 0x7f50f4137ce0 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 21863577 0 0) 0x7f50f40a2110 con 0x7f50f413e5f0 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient(hunting): my global_id is 84551 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50f40a6420 con 0x7f50f413e5f0 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2290587633 0 0) 0x7f50e0003530 con 0x7f50f4137ce0 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 >> v1:172.21.15.78:6790/0 conn(0x7f50f413e5f0 legacy=0x7f50f41409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 >> v1:172.21.15.146:6789/0 conn(0x7f50f40a9610 legacy=0x7f50f413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 monclient: found mon.noname-a 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f40a4bc0 con 0x7f50f4137ce0 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f50e40016a0 con 0x7f50f4137ce0 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 5 monclient: authenticate success, global_id 114111 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: dump: 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.783 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50e40018c0 con 0x7f50f4137ce0 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 1 -- 172.21.15.78:0/4201547052 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f50e4001c90 con 0x7f50f4137ce0 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 4 set_mon_vals no callback set 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50ebfff640 10 monclient: dump: 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.784 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.785 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.788 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.788 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.788 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: shutdown 2024-09-15T20:41:14.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- 172.21.15.78:0/4201547052 >> v1:172.21.15.78:6789/0 conn(0x7f50f4137ce0 legacy=0x7f50f41380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- 172.21.15.78:0/4201547052 shutdown_connections 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- 172.21.15.78:0/4201547052 >> 172.21.15.78:0/4201547052 conn(0x7f50f40aa6c0 msgr2=0x7f50f40aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- 172.21.15.78:0/4201547052 shutdown_connections 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 1 -- 172.21.15.78:0/4201547052 wait complete. 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: build_initial_monmap 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.769+0000 7f50fb66e640 10 monclient: monmap: 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:14.789 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 librados: starting msgr at 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 librados: starting objecter 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 Processor -- start 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 -- start start 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 librados: setting wanted keys 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 librados: calling monclient init 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient: init 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:14.790 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient(hunting): picked mon.noname-b con 0x7f50f40a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient(hunting): picked mon.noname-c con 0x7f50f4137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient(hunting): picked mon.noname-a con 0x7f50f413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f40a3db0 con 0x7f50f413e5f0 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f4138f70 con 0x7f50f40a9610 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50f409e7f0 con 0x7f50f4137ce0 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient(hunting): authenticate will time out at 1735.660812s 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50f9c0b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f50f413e5f0 0x7f50f41d52b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42288/0 (socket says 172.21.15.78:42288) 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50f9c0b640 1 -- 172.21.15.78:0/1309905308 learned_addr learned my addr 172.21.15.78:0/1309905308 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.?.objecter ms_handle_connect 0x7f50f413e5f0 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.?.objecter ms_handle_connect 0x7f50f4137ce0 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.791 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3365576619 0 0) 0x7f50f40a3db0 con 0x7f50f413e5f0 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): my global_id is 114114 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d8004ec0 con 0x7f50f413e5f0 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4055595092 0 0) 0x7f50f409e7f0 con 0x7f50f4137ce0 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): my global_id is 84554 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50f40a3db0 con 0x7f50f4137ce0 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.?.objecter ms_handle_connect 0x7f50f40a9610 2024-09-15T20:41:14.792 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3447721912 0 0) 0x7f50d8004ec0 con 0x7f50f413e5f0 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50f409e7f0 con 0x7f50f413e5f0 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f50ec001c50 con 0x7f50f413e5f0 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): dump: 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.793 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2998844382 0 0) 0x7f50f4138f70 con 0x7f50f40a9610 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): my global_id is 23602 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d8004ec0 con 0x7f50f40a9610 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 529631750 0 0) 0x7f50f409e7f0 con 0x7f50f413e5f0 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 >> v1:172.21.15.78:6790/0 conn(0x7f50f4137ce0 legacy=0x7f50f41cd8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 >> v1:172.21.15.146:6789/0 conn(0x7f50f40a9610 legacy=0x7f50f41cd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 monclient: found mon.a 2024-09-15T20:41:14.794 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50f40a0570 con 0x7f50f413e5f0 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 5 monclient: authenticate success, global_id 114114 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient: _renew_subs 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f50f409f0b0 con 0x7f50f413e5f0 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 client.114114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient: _renew_subs 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f50f4139b20 con 0x7f50f413e5f0 2024-09-15T20:41:14.795 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 librados: init done 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f50ec0039a0 con 0x7f50f413e5f0 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f50ec003d70 con 0x7f50f413e5f0 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: dump: 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.796 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50eb7fe640 4 set_mon_vals no callback set 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f50d40065d0 con 0x7f50d4002870 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f50ec003f10 con 0x7f50f413e5f0 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.114114.objecter ms_handle_connect 0x7f50d4002870 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.114114.objecter resend_mon_ops 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(379..379 src has 223..379) ==== 5346+0+0 (unknown 1175008561 0 0) 0x7f50ec04ba80 con 0x7f50f413e5f0 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.114114.objecter ms_dispatch 0x7f50f4000ba0 osd_map(379..379 src has 223..379) 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 3 client.114114.objecter handle_osd_map got epochs [379,379] > 0 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 3 client.114114.objecter handle_osd_map decoding full epoch 379 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 20 client.114114.objecter dump_active .. 0 homeless 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3044325118 0 0) 0x7f50d40065d0 con 0x7f50d4002870 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient(hunting): my global_id is 84557 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50f409e7f0 con 0x7f50d4002870 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50f8c09640 10 monclient: get_auth_request con 0x7f50d803df00 auth_method 0 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2446038623 0 0) 0x7f50f409e7f0 con 0x7f50d4002870 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50d40065d0 con 0x7f50d4002870 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f50e40016a0 con 0x7f50d4002870 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 monclient: dump: 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.799 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3806626736 0 0) 0x7f50d40065d0 con 0x7f50d4002870 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f50d400a170 con 0x7f50d4002870 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.773+0000 7f50e9ffb640 10 client.114114.objecter ms_handle_connect 0x7f50d803df00 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.777+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f50e4001a20 con 0x7f50d4002870 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.777+0000 7f50e9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.777+0000 7f50e9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.777+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 >> v1:172.21.15.78:6790/0 conn(0x7f50d4002870 legacy=0x7f50d4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.777+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 >> v1:172.21.15.78:6790/0 conn(0x7f50d4002870 legacy=0x7f50d4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:41:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f50d4009e60 con 0x7f50d4009120 2024-09-15T20:41:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 client.114114.objecter ms_handle_connect 0x7f50d4009120 2024-09-15T20:41:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 client.114114.objecter resend_mon_ops 2024-09-15T20:41:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 869904245 0 0) 0x7f50d4009e60 con 0x7f50d4009120 2024-09-15T20:41:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient(hunting): my global_id is 84560 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50d40065d0 con 0x7f50d4009120 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 646531924 0 0) 0x7f50d40065d0 con 0x7f50d4009120 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f50d4009e60 con 0x7f50d4009120 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f50e4004af0 con 0x7f50d4009120 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient: dump: 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.803 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 626137199 0 0) 0x7f50d4009e60 con 0x7f50d4009120 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f50d400c400 con 0x7f50d4009120 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 446488271) 0x7f50e4004ae0 con 0x7f50d4009120 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 >> v1:172.21.15.78:6790/0 conn(0x7f50d4009120 legacy=0x7f50d40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50e9ffb640 1 -- 172.21.15.78:0/1309905308 >> v1:172.21.15.78:6790/0 conn(0x7f50d4009120 legacy=0x7f50d40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 38, 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 346468, 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:41:14.805 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.806 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.807 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:41:14.808 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:41:14.809 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:41:14.810 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.811 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.812 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:14.813 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.814 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.815 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.816 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:41:14.817 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:41:14.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 10 librados: watch_flush enter 2024-09-15T20:41:14.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 10 librados: watch_flush exit 2024-09-15T20:41:14.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 20 client.114114.objecter shutdown clearing up homeless session... 2024-09-15T20:41:14.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 10 client.114114.objecter successfully canceled tick 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 >> v1:172.21.15.146:6812/435598642 conn(0x7f50d803df00 legacy=0x7f50d80403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 10 monclient: shutdown 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 >> v1:172.21.15.78:6789/0 conn(0x7f50f413e5f0 legacy=0x7f50f41d52b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50f9c0b640 1 -- 172.21.15.78:0/1309905308 reap_dead start 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50f9c0b640 1 -- 172.21.15.78:0/1309905308 reap_dead start 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 shutdown_connections 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 >> 172.21.15.78:0/1309905308 conn(0x7f50f40aa6c0 msgr2=0x7f50f40a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 shutdown_connections 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 1 -- 172.21.15.78:0/1309905308 wait complete. 2024-09-15T20:41:14.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.781+0000 7f50fb66e640 1 librados: shutdown 2024-09-15T20:41:14.820 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:41:14.820 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:41:14.820 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:41:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.929+0000 7fefc762e640 10 monclient: get_monmap_and_config 2024-09-15T20:41:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.929+0000 7fefc762e640 10 monclient: build_initial_monmap 2024-09-15T20:41:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.929+0000 7fefc762e640 10 monclient: monmap: 2024-09-15T20:41:14.935 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:14.936597+0000 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:14.936597+0000 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:14.936 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 Processor -- start 2024-09-15T20:41:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- start start 2024-09-15T20:41:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient: init 2024-09-15T20:41:14.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient(hunting): picked mon.noname-b con 0x7fefc0137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient(hunting): picked mon.noname-a con 0x7fefc0074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient(hunting): picked mon.noname-c con 0x7fefc013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc0138850 con 0x7fefc0074500 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc00a5da0 con 0x7fefc013e370 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc0073560 con 0x7fefc0137d60 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient(hunting): authenticate will time out at 1735.821277s 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc4bc9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fefc0074500 0x7fefc013cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42298/0 (socket says 172.21.15.78:42298) 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc4bc9640 1 -- 172.21.15.78:0/1441297900 learned_addr learned my addr 172.21.15.78:0/1441297900 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1413048720 0 0) 0x7fefc0138850 con 0x7fefc0074500 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): my global_id is 114117 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefac003530 con 0x7fefc0074500 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 219670290 0 0) 0x7fefc0073560 con 0x7fefc0137d60 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): my global_id is 84563 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefc0138850 con 0x7fefc0137d60 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 702898790 0 0) 0x7fefac003530 con 0x7fefc0074500 2024-09-15T20:41:14.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 >> v1:172.21.15.78:6790/0 conn(0x7fefc0137d60 legacy=0x7fefc0138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 >> v1:172.21.15.146:6789/0 conn(0x7fefc013e370 legacy=0x7fefc0140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 monclient: found mon.noname-a 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc0073840 con 0x7fefc0074500 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fefb00016a0 con 0x7fefc0074500 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 5 monclient: authenticate success, global_id 114117 2024-09-15T20:41:14.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: dump: 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.950 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc4bc9640 1 -- 172.21.15.78:0/1441297900 >> v1:172.21.15.78:6789/0 conn(0x7fefc0074500 legacy=0x7fefc013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc4bc9640 1 -- 172.21.15.78:0/1441297900 >> v1:172.21.15.78:6789/0 conn(0x7fefc0074500 legacy=0x7fefc013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc4bc9640 1 --1- 172.21.15.78:0/1441297900 >> v1:172.21.15.78:6789/0 conn(0x7fefc0074500 0x7fefc013cc50 :-1 s=OPENED pgs=9012 cs=1 l=1).handle_message read tag failed 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc4bc9640 1 --1- 172.21.15.78:0/1441297900 >> v1:172.21.15.78:6789/0 conn(0x7fefc0074500 0x7fefc013cc50 :-1 s=OPENED pgs=9012 cs=1 l=1).fault on lossy channel, failing 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: ms_handle_reset current mon v1:172.21.15.78:6789/0 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 >> v1:172.21.15.78:6789/0 conn(0x7fefc0074500 legacy=0x7fefc013cc50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): picked mon.b con 0x7fefc013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): picked mon.c con 0x7fefc0137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): picked mon.a con 0x7fefac00b0e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fefac003530 con 0x7fefac00b0e0 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fefc0138850 con 0x7fefc013e370 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fefc0073560 con 0x7fefc0137d60 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.78:6789/0 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2295076722 0 0) 0x7fefc0073560 con 0x7fefc0137d60 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.78:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fefac010e80 con 0x7fefc0137d60 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1641557027 0 0) 0x7fefac003530 con 0x7fefac00b0e0 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.78:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7fefc0073560 con 0x7fefac00b0e0 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 3042719277 0 0) 0x7fefac010e80 con 0x7fefc0137d60 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 >> v1:172.21.15.146:6789/0 conn(0x7fefc013e370 legacy=0x7fefac004ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 >> v1:172.21.15.78:6789/0 conn(0x7fefac00b0e0 legacy=0x7fefac00d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 monclient: found mon.c 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=2+}) -- 0x7fefac012010 con 0x7fefc0137d60 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fefb00048b0 con 0x7fefc0137d60 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: dump: 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 1 -- 172.21.15.78:0/1441297900 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fefb00048b0 con 0x7fefc0137d60 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefb7fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 4 set_mon_vals no callback set 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 10 monclient: shutdown 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- 172.21.15.78:0/1441297900 >> v1:172.21.15.78:6790/0 conn(0x7fefc0137d60 legacy=0x7fefac0099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- 172.21.15.78:0/1441297900 shutdown_connections 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- 172.21.15.78:0/1441297900 >> 172.21.15.78:0/1441297900 conn(0x7fefc00aa650 msgr2=0x7fefc00aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- 172.21.15.78:0/1441297900 shutdown_connections 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.933+0000 7fefc762e640 1 -- 172.21.15.78:0/1441297900 wait complete. 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: build_initial_monmap 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: monmap: 2024-09-15T20:41:14.955 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 librados: starting msgr at 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 librados: starting objecter 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 Processor -- start 2024-09-15T20:41:14.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 -- start start 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 librados: setting wanted keys 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 librados: calling monclient init 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: init 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient(hunting): picked mon.noname-b con 0x7fefac00b0e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient(hunting): picked mon.noname-c con 0x7fefc0137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient(hunting): picked mon.noname-a con 0x7fefc013e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc0121d80 con 0x7fefc013e370 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc00a3ac0 con 0x7fefac00b0e0 2024-09-15T20:41:14.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fefc00a9900 con 0x7fefc0137d60 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient(hunting): authenticate will time out at 1735.825173s 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc4bc9640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fefc0137d60 0x7fefc01d1730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35374/0 (socket says 172.21.15.78:35374) 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc5bcb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fefc013e370 0x7fefc01d4e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42318/0 (socket says 172.21.15.78:42318) 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc4bc9640 1 -- 172.21.15.78:0/870646688 learned_addr learned my addr 172.21.15.78:0/870646688 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.?.objecter ms_handle_connect 0x7fefc0137d60 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.?.objecter ms_handle_connect 0x7fefc013e370 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 733719963 0 0) 0x7fefc00a9900 con 0x7fefc0137d60 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): my global_id is 84566 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefa4004ec0 con 0x7fefc0137d60 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.?.objecter ms_handle_connect 0x7fefac00b0e0 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2164143491 0 0) 0x7fefc0121d80 con 0x7fefc013e370 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): my global_id is 114120 2024-09-15T20:41:14.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefc00a9900 con 0x7fefc013e370 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2208950240 0 0) 0x7fefa4004ec0 con 0x7fefc0137d60 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefc0121d80 con 0x7fefc0137d60 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fefb0001660 con 0x7fefc0137d60 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): dump: 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.960 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.961 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:14.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2618628155 0 0) 0x7fefc00a9900 con 0x7fefc013e370 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefa4004ec0 con 0x7fefc013e370 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3054905780 0 0) 0x7fefc00a3ac0 con 0x7fefac00b0e0 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): my global_id is 23608 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefc00a9900 con 0x7fefac00b0e0 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fefb8001c00 con 0x7fefc013e370 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): dump: 2024-09-15T20:41:14.962 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 289518763 0 0) 0x7fefa4004ec0 con 0x7fefc013e370 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 >> v1:172.21.15.78:6790/0 conn(0x7fefc0137d60 legacy=0x7fefc01d1730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 >> v1:172.21.15.146:6789/0 conn(0x7fefac00b0e0 legacy=0x7fefc01cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 monclient: found mon.a 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefc009e0b0 con 0x7fefc013e370 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient: _finish_auth 0 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 5 monclient: authenticate success, global_id 114120 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: _renew_subs 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fefc0121aa0 con 0x7fefc013e370 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fefb8003980 con 0x7fefc013e370 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:14.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 client.114120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fefb8003dd0 con 0x7fefc013e370 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: _renew_subs 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fefc0122480 con 0x7fefc013e370 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 librados: init done 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb77fe640 4 set_mon_vals no callback set 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient: dump: 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.965 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fefc00008e0 con 0x7fefc013cc90 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.114120.objecter ms_handle_connect 0x7fefc013cc90 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.114120.objecter resend_mon_ops 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fefb8003f70 con 0x7fefc013e370 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(379..379 src has 223..379) ==== 5346+0+0 (unknown 1175008561 0 0) 0x7fefb804b870 con 0x7fefc013e370 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 client.114120.objecter ms_dispatch 0x7fefc0000ba0 osd_map(379..379 src has 223..379) 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 3 client.114120.objecter handle_osd_map got epochs [379,379] > 0 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 3 client.114120.objecter handle_osd_map decoding full epoch 379 2024-09-15T20:41:14.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 20 client.114120.objecter dump_active .. 0 homeless 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165764 0 0) 0x7fefc00008e0 con 0x7fefc013cc90 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 10 monclient(hunting): my global_id is 23611 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.937+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefa4004ec0 con 0x7fefc013cc90 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefc4bc9640 10 monclient: get_auth_request con 0x7fefa403e1d0 auth_method 0 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3072648635 0 0) 0x7fefa4004ec0 con 0x7fefc013cc90 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefc00008e0 con 0x7fefc013cc90 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fefbc001080 con 0x7fefc013cc90 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 10 monclient: dump: 2024-09-15T20:41:14.967 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 10 client.114120.objecter ms_handle_connect 0x7fefa403e1d0 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3588122780 0 0) 0x7fefc00008e0 con 0x7fefc013cc90 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fefc013d070 con 0x7fefc013cc90 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fefbc001c80 con 0x7fefc013cc90 2024-09-15T20:41:14.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:14.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:14.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 >> v1:172.21.15.146:6789/0 conn(0x7fefc013cc90 legacy=0x7fefc0138260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.941+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 >> v1:172.21.15.146:6789/0 conn(0x7fefc013cc90 legacy=0x7fefc0138260 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.945+0000 7fefc762e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.945+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fefc01cd2b0 con 0x7fefc01cce60 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 client.114120.objecter ms_handle_connect 0x7fefc01cce60 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 client.114120.objecter resend_mon_ops 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3169728055 0 0) 0x7fefc01cd2b0 con 0x7fefc01cce60 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient(hunting): my global_id is 23614 2024-09-15T20:41:14.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fefc00008e0 con 0x7fefc01cce60 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1055452934 0 0) 0x7fefc00008e0 con 0x7fefc01cce60 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fefc01cd2b0 con 0x7fefc01cce60 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fefbc0037a0 con 0x7fefc01cce60 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient: dump: 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:14.971 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 868121143 0 0) 0x7fefc01cd2b0 con 0x7fefc01cce60 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fefc009cfc0 con 0x7fefc01cce60 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 1503818047) 0x7fefbc003730 con 0x7fefc01cce60 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 >> v1:172.21.15.146:6789/0 conn(0x7fefc01cce60 legacy=0x7fefc0138970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefb5ffb640 1 -- 172.21.15.78:0/870646688 >> v1:172.21.15.146:6789/0 conn(0x7fefc01cce60 legacy=0x7fefc0138970 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:41:14.972 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:41:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 38, 2024-09-15T20:41:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 587457, 2024-09-15T20:41:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:41:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:41:14.973 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:41:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:41:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:41:14.974 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:41:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:41:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.975 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.976 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:41:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:41:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.977 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.978 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:41:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:41:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:41:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:41:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:41:14.979 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:41:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:41:14.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:41:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:41:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:41:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:41:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:41:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:41:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:14.981 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:14.982 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:41:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.985 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:41:14.985 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:41:14.985 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.985 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.985 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:14.987 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:14.988 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.989 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:41:14.990 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:14.991 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:14.992 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:41:14.992 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:41:14.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 10 librados: watch_flush enter 2024-09-15T20:41:14.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 10 librados: watch_flush exit 2024-09-15T20:41:14.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 20 client.114120.objecter shutdown clearing up homeless session... 2024-09-15T20:41:14.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 10 client.114120.objecter successfully canceled tick 2024-09-15T20:41:14.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 >> v1:172.21.15.146:6812/435598642 conn(0x7fefa403e1d0 legacy=0x7fefa4040690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 10 monclient: shutdown 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 >> v1:172.21.15.78:6789/0 conn(0x7fefc013e370 legacy=0x7fefc01d4e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc5bcb640 1 -- 172.21.15.78:0/870646688 reap_dead start 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc5bcb640 1 -- 172.21.15.78:0/870646688 reap_dead start 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 shutdown_connections 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 >> 172.21.15.78:0/870646688 conn(0x7fefc00aa650 msgr2=0x7fefc00a8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 shutdown_connections 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 1 -- 172.21.15.78:0/870646688 wait complete. 2024-09-15T20:41:14.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:14.949+0000 7fefc762e640 1 librados: shutdown 2024-09-15T20:41:14.995 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:41:14.996 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:41:14.996 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:41:15.110 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.105+0000 7f64c4c85640 10 monclient: get_monmap_and_config 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.105+0000 7f64c4c85640 10 monclient: build_initial_monmap 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.105+0000 7f64c4c85640 10 monclient: monmap: 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:15.112109+0000 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:15.112109+0000 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:15.111 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:15.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.105+0000 7f64c4c85640 1 Processor -- start 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- start start 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: init 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): picked mon.noname-c con 0x7f64c0135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): picked mon.noname-b con 0x7f64c00a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): picked mon.noname-a con 0x7f64c013c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c00745e0 con 0x7f64c013c220 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c006ecf0 con 0x7f64c0135b60 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c006efd0 con 0x7f64c00a24e0 2024-09-15T20:41:15.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): authenticate will time out at 1735.996783s 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f64c013c220 0x7f64c013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42332/0 (socket says 172.21.15.78:42332) 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bed9d640 1 -- 172.21.15.78:0/4053144683 learned_addr learned my addr 172.21.15.78:0/4053144683 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1529608508 0 0) 0x7f64c00745e0 con 0x7f64c013c220 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient(hunting): _negotiate 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient(hunting): my global_id is 114123 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64ac003530 con 0x7f64c013c220 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2832385292 0 0) 0x7f64c006efd0 con 0x7f64c00a24e0 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient(hunting): _negotiate 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient(hunting): my global_id is 84569 2024-09-15T20:41:15.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c00745e0 con 0x7f64c00a24e0 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2580501042 0 0) 0x7f64c00745e0 con 0x7f64c00a24e0 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 >> v1:172.21.15.146:6789/0 conn(0x7f64c0135b60 legacy=0x7f64c0135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 >> v1:172.21.15.78:6789/0 conn(0x7f64c013c220 legacy=0x7f64c013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: _finish_hunting 0 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 monclient: found mon.noname-b 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c0136650 con 0x7f64c00a24e0 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: _finish_auth 0 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: _check_auth_tickets 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f64b8001a00 con 0x7f64c00a24e0 2024-09-15T20:41:15.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: dump: 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:15.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 5 monclient: authenticate success, global_id 84569 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64b8001ba0 con 0x7f64c00a24e0 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 1 -- 172.21.15.78:0/4053144683 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f64b8001fb0 con 0x7f64c00a24e0 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bd59a640 10 monclient: dump: 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:15.124 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 4 set_mon_vals no callback set 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: shutdown 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- 172.21.15.78:0/4053144683 >> v1:172.21.15.78:6790/0 conn(0x7f64c00a24e0 legacy=0x7f64c013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- 172.21.15.78:0/4053144683 shutdown_connections 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- 172.21.15.78:0/4053144683 >> 172.21.15.78:0/4053144683 conn(0x7f64c00a63c0 msgr2=0x7f64c00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:15.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- 172.21.15.78:0/4053144683 shutdown_connections 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- 172.21.15.78:0/4053144683 wait complete. 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: build_initial_monmap 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: monmap: 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:15.126 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 librados: starting msgr at 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 librados: starting objecter 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 Processor -- start 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- start start 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 librados: setting wanted keys 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 librados: calling monclient init 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: init 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:15.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): picked mon.noname-a con 0x7f64c00a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): picked mon.noname-c con 0x7f64c0135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): picked mon.noname-b con 0x7f64c013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c009e770 con 0x7f64c00a24e0 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c00729d0 con 0x7f64c013c220 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64c0073860 con 0x7f64c0135b60 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64c4c85640 10 monclient(hunting): authenticate will time out at 1735.999834s 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64be59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f64c00a24e0 0x7f64c0071bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42344/0 (socket says 172.21.15.78:42344) 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64be59c640 1 -- 172.21.15.78:0/1267208996 learned_addr learned my addr 172.21.15.78:0/1267208996 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.109+0000 7f64bdd9b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f64c0135b60 0x7f64c00722c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35398/0 (socket says 172.21.15.78:35398) 2024-09-15T20:41:15.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.?.objecter ms_handle_connect 0x7f64c0135b60 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.?.objecter ms_handle_connect 0x7f64c00a24e0 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 361489306 0 0) 0x7f64c0073860 con 0x7f64c0135b60 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): my global_id is 84572 2024-09-15T20:41:15.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64a0004ec0 con 0x7f64c0135b60 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565604656 0 0) 0x7f64c009e770 con 0x7f64c00a24e0 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): my global_id is 114126 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c0073860 con 0x7f64c00a24e0 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.?.objecter ms_handle_connect 0x7f64c013c220 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:15.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3787622380 0 0) 0x7f64a0004ec0 con 0x7f64c0135b60 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64c009e770 con 0x7f64c0135b60 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f64b8001900 con 0x7f64c0135b60 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): dump: 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:15.131 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3185800351 0 0) 0x7f64c0073860 con 0x7f64c00a24e0 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f64a0004ec0 con 0x7f64c00a24e0 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f64a8001900 con 0x7f64c00a24e0 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): dump: 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:15.132 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393725981 0 0) 0x7f64c00729d0 con 0x7f64c013c220 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:15.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): my global_id is 23620 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f64c0073860 con 0x7f64c013c220 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 989287300 0 0) 0x7f64a0004ec0 con 0x7f64c00a24e0 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.78:6790/0 conn(0x7f64c0135b60 legacy=0x7f64c00722c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.146:6789/0 conn(0x7f64c013c220 legacy=0x7f64c01d5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 monclient: found mon.a 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c0073620 con 0x7f64c00a24e0 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _finish_auth 0 2024-09-15T20:41:15.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 5 monclient: authenticate success, global_id 114126 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 10 monclient: _renew_subs 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f64c009d6a0 con 0x7f64c00a24e0 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 10 client.114126.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 10 monclient: _renew_subs 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f64c0073160 con 0x7f64c00a24e0 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 1 librados: init done 2024-09-15T20:41:15.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64be59c640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.78:6789/0 conn(0x7f64c00a24e0 legacy=0x7f64c0071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64be59c640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.78:6789/0 conn(0x7f64c00a24e0 legacy=0x7f64c0071bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64be59c640 1 --1- 172.21.15.78:0/1267208996 >> v1:172.21.15.78:6789/0 conn(0x7f64c00a24e0 0x7f64c0071bb0 :-1 s=OPENED pgs=9016 cs=1 l=1).handle_message read tag failed 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64be59c640 1 --1- 172.21.15.78:0/1267208996 >> v1:172.21.15.78:6789/0 conn(0x7f64c00a24e0 0x7f64c0071bb0 :-1 s=OPENED pgs=9016 cs=1 l=1).fault on lossy channel, failing 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: ms_handle_reset current mon v1:172.21.15.78:6789/0 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.78:6789/0 conn(0x7f64c00a24e0 legacy=0x7f64c0071bb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): picked mon.a con 0x7f64a00056d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): picked mon.b con 0x7f64c013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): picked mon.c con 0x7f64c0135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f64a0004ec0 con 0x7f64a00056d0 2024-09-15T20:41:15.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f64c009e770 con 0x7f64c013c220 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f64c00729d0 con 0x7f64c0135b60 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.114126.objecter ms_handle_connect 0x7f64c0135b60 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.114126.objecter resend_mon_ops 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.114126.objecter ms_handle_connect 0x7f64a00056d0 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.114126.objecter resend_mon_ops 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 955583716 0 0) 0x7f64c00729d0 con 0x7f64c0135b60 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f64a0011c80 con 0x7f64c0135b60 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3928977280 0 0) 0x7f64a0004ec0 con 0x7f64a00056d0 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:15.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f64c00729d0 con 0x7f64a00056d0 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.114126.objecter ms_handle_connect 0x7f64c013c220 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.114126.objecter resend_mon_ops 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 4112685352 0 0) 0x7f64a0011c80 con 0x7f64c0135b60 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.146:6789/0 conn(0x7f64c013c220 legacy=0x7f64a000ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.78:6789/0 conn(0x7f64a00056d0 legacy=0x7f64a0009340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 monclient: found mon.c 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) -- 0x7f64a0014fd0 con 0x7f64c0135b60 2024-09-15T20:41:15.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _finish_auth 0 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f64b0000ff0 con 0x7f64c0135b60 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: dump: 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:15.139 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:15.140 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:15.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f64b0001190 con 0x7f64c0135b60 2024-09-15T20:41:15.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:15.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:15.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:15.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:15.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64bcd99640 4 set_mon_vals no callback set 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64c012fa90 con 0x7f64c0135b60 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f64b0001400 con 0x7f64c0135b60 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 6 ==== osd_map(379..379 src has 223..379) ==== 5346+0+0 (unknown 1175008561 0 0) 0x7f64b004ad70 con 0x7f64c0135b60 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.114126.objecter ms_dispatch 0x7f64c0000ba0 osd_map(379..379 src has 223..379) 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 3 client.114126.objecter handle_osd_map got epochs [379,379] > 0 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 3 client.114126.objecter handle_osd_map decoding full epoch 379 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 20 client.114126.objecter dump_active .. 0 homeless 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64be59c640 10 monclient: get_auth_request con 0x7f64c00a24e0 auth_method 0 2024-09-15T20:41:15.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.113+0000 7f64b6ffd640 10 client.114126.objecter ms_handle_connect 0x7f64c00a24e0 2024-09-15T20:41:15.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.117+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f64b0016070 con 0x7f64c0135b60 2024-09-15T20:41:15.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.117+0000 7f64b6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:15.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.117+0000 7f64b6ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:15.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.353+0000 7f64c4c85640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:41:15.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.353+0000 7f64c4c85640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:41:15.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.353+0000 7f64c4c85640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:15.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.353+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f64c009dec0 con 0x7f64c0135b60 2024-09-15T20:41:15.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.357+0000 7f64b6ffd640 1 -- 172.21.15.78:0/1267208996 <== mon.2 v1:172.21.15.78:6790/0 8 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f64b0015970 con 0x7f64c0135b60 2024-09-15T20:41:15.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.357+0000 7f64b6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:41:15.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.357+0000 7f64b6ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:15.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.357+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.357+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=11,logm=58} crc {auth=3848354109,config=2586702517,health=3928057762,kv=3677150869,logm=3576742565}) 2024-09-15T20:41:15.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.357+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=18,config=2,health=12,kv=11,logm=57} crc {auth=3117613400,config=2586702516,health=3623406937,kv=1022021693,logm=4259939086}) 2024-09-15T20:41:15.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.357+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.357+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=18,config=2,health=11,kv=11,logm=58} crc {auth=3848354109,config=2586702517,health=3928057762,kv=3677150869,logm=3576742565}) 2024-09-15T20:41:15.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.357+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=17,config=2,health=12,kv=12,logm=57} crc {auth=4032504237,config=2586702516,health=3623406937,kv=1721617520,logm=880127127}) 2024-09-15T20:41:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 10 librados: watch_flush enter 2024-09-15T20:41:15.365 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.365 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3898692976}) 2024-09-15T20:41:15.365 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2910666332}) 2024-09-15T20:41:15.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 10 librados: watch_flush exit 2024-09-15T20:41:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 20 client.114126.objecter shutdown clearing up homeless session... 2024-09-15T20:41:15.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 10 client.114126.objecter successfully canceled tick 2024-09-15T20:41:15.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3898692976}) 2024-09-15T20:41:15.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2492545128}) 2024-09-15T20:41:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.146:6812/435598642 conn(0x7f64c00a24e0 legacy=0x7f64a004b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 10 monclient: shutdown 2024-09-15T20:41:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 >> v1:172.21.15.78:6790/0 conn(0x7f64c0135b60 legacy=0x7f64a00101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:15.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 shutdown_connections 2024-09-15T20:41:15.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 >> 172.21.15.78:0/1267208996 conn(0x7f64c00a63c0 msgr2=0x7f64c012ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:15.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 shutdown_connections 2024-09-15T20:41:15.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 1 -- 172.21.15.78:0/1267208996 wait complete. 2024-09-15T20:41:15.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:15.361+0000 7f64c4c85640 1 librados: shutdown 2024-09-15T20:41:15.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165260320}) 2024-09-15T20:41:15.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=654819611}) 2024-09-15T20:41:15.369 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.369 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=165260320}) 2024-09-15T20:41:15.369 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.361+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2966358552}) 2024-09-15T20:41:15.369 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.369 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558047432}) 2024-09-15T20:41:15.369 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=155687861}) 2024-09-15T20:41:15.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2558047432}) 2024-09-15T20:41:15.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=628010006}) 2024-09-15T20:41:15.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612110587}) 2024-09-15T20:41:15.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=781849130}) 2024-09-15T20:41:15.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2612110587}) 2024-09-15T20:41:15.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.365+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3996153838}) 2024-09-15T20:41:15.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284096399}) 2024-09-15T20:41:15.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244273966}) 2024-09-15T20:41:15.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4284096399}) 2024-09-15T20:41:15.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3701676984}) 2024-09-15T20:41:15.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568427891}) 2024-09-15T20:41:15.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1059423351}) 2024-09-15T20:41:15.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3568427891}) 2024-09-15T20:41:15.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.369+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1272281370}) 2024-09-15T20:41:15.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3865591797}) 2024-09-15T20:41:15.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2919604137}) 2024-09-15T20:41:15.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3865591797}) 2024-09-15T20:41:15.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2860956702}) 2024-09-15T20:41:15.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536260782}) 2024-09-15T20:41:15.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=950364269}) 2024-09-15T20:41:15.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3536260782}) 2024-09-15T20:41:15.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.373+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3299461717}) 2024-09-15T20:41:15.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022758683}) 2024-09-15T20:41:15.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1421064228}) 2024-09-15T20:41:15.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4022758683}) 2024-09-15T20:41:15.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2340702282}) 2024-09-15T20:41:15.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858504817}) 2024-09-15T20:41:15.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=800763286}) 2024-09-15T20:41:15.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3858504817}) 2024-09-15T20:41:15.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.377+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=909225592}) 2024-09-15T20:41:15.384 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:41:15.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3670311425}) 2024-09-15T20:41:15.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1093330275}) 2024-09-15T20:41:15.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3670311425}) 2024-09-15T20:41:15.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2759931041}) 2024-09-15T20:41:15.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697332756}) 2024-09-15T20:41:15.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2495685880}) 2024-09-15T20:41:15.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2697332756}) 2024-09-15T20:41:15.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4147669183}) 2024-09-15T20:41:15.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261792460}) 2024-09-15T20:41:15.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2524403305}) 2024-09-15T20:41:15.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2261792460}) 2024-09-15T20:41:15.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.381+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=579262099}) 2024-09-15T20:41:15.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601080163}) 2024-09-15T20:41:15.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=675644715}) 2024-09-15T20:41:15.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601080163}) 2024-09-15T20:41:15.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2083193957}) 2024-09-15T20:41:15.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391498959}) 2024-09-15T20:41:15.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2221482243}) 2024-09-15T20:41:15.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3391498959}) 2024-09-15T20:41:15.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.385+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=337607053}) 2024-09-15T20:41:15.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397202106}) 2024-09-15T20:41:15.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3972900899}) 2024-09-15T20:41:15.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=397202106}) 2024-09-15T20:41:15.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=582177284}) 2024-09-15T20:41:15.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962249838}) 2024-09-15T20:41:15.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3597922529}) 2024-09-15T20:41:15.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3962249838}) 2024-09-15T20:41:15.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.389+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556614726}) 2024-09-15T20:41:15.396 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344285559}) 2024-09-15T20:41:15.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2642383315}) 2024-09-15T20:41:15.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2344285559}) 2024-09-15T20:41:15.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3484874447}) 2024-09-15T20:41:15.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332831496}) 2024-09-15T20:41:15.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=490976973}) 2024-09-15T20:41:15.399 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.399 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332831496}) 2024-09-15T20:41:15.399 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.393+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2384600856}) 2024-09-15T20:41:15.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149282076}) 2024-09-15T20:41:15.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3274107732}) 2024-09-15T20:41:15.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.401 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3149282076}) 2024-09-15T20:41:15.401 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3866834904}) 2024-09-15T20:41:15.406 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.406 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296356995}) 2024-09-15T20:41:15.406 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2758634960}) 2024-09-15T20:41:15.406 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.406 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2296356995}) 2024-09-15T20:41:15.406 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.397+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4087320999}) 2024-09-15T20:41:15.406 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.406 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=566832838}) 2024-09-15T20:41:15.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=547256707}) 2024-09-15T20:41:15.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=566832838}) 2024-09-15T20:41:15.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3329869129}) 2024-09-15T20:41:15.408 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.408 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943589315}) 2024-09-15T20:41:15.408 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3014786430}) 2024-09-15T20:41:15.408 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.408 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1943589315}) 2024-09-15T20:41:15.408 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.401+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=419988492}) 2024-09-15T20:41:15.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.405+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.405+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996040495}) 2024-09-15T20:41:15.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.405+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2122712203}) 2024-09-15T20:41:15.410 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.405+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.410 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.405+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996040495}) 2024-09-15T20:41:15.410 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.405+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=209985023}) 2024-09-15T20:41:15.412 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.412 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096758330}) 2024-09-15T20:41:15.412 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=475943572}) 2024-09-15T20:41:15.412 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.412 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3096758330}) 2024-09-15T20:41:15.412 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2742965153}) 2024-09-15T20:41:15.414 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.414 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226067562}) 2024-09-15T20:41:15.414 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3573855545}) 2024-09-15T20:41:15.414 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.414 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2226067562}) 2024-09-15T20:41:15.414 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.409+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2556948174}) 2024-09-15T20:41:15.416 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.416 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2701941668}) 2024-09-15T20:41:15.416 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1206798133}) 2024-09-15T20:41:15.416 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.416 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2701941668}) 2024-09-15T20:41:15.416 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=533759363}) 2024-09-15T20:41:15.418 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.418 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220115194}) 2024-09-15T20:41:15.419 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4034330204}) 2024-09-15T20:41:15.419 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.419 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220115194}) 2024-09-15T20:41:15.419 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.413+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2394531094}) 2024-09-15T20:41:15.420 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136700014}) 2024-09-15T20:41:15.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2866848307}) 2024-09-15T20:41:15.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=136700014}) 2024-09-15T20:41:15.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=185517639}) 2024-09-15T20:41:15.422 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.422 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981113374}) 2024-09-15T20:41:15.422 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1341861451}) 2024-09-15T20:41:15.422 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.423 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3981113374}) 2024-09-15T20:41:15.423 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.417+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=702336680}) 2024-09-15T20:41:15.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661135512}) 2024-09-15T20:41:15.424 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4268316574}) 2024-09-15T20:41:15.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2661135512}) 2024-09-15T20:41:15.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2406526057}) 2024-09-15T20:41:15.426 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409463202}) 2024-09-15T20:41:15.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=750342603}) 2024-09-15T20:41:15.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1409463202}) 2024-09-15T20:41:15.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.421+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2861333224}) 2024-09-15T20:41:15.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808752233}) 2024-09-15T20:41:15.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3953877514}) 2024-09-15T20:41:15.430 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.430 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3808752233}) 2024-09-15T20:41:15.430 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1744557754}) 2024-09-15T20:41:15.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3324701253}) 2024-09-15T20:41:15.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3855460854}) 2024-09-15T20:41:15.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.425+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.429+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3324701253}) 2024-09-15T20:41:15.432 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.429+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1353052280}) 2024-09-15T20:41:15.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.429+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.429+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742921142}) 2024-09-15T20:41:15.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.429+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=698710419}) 2024-09-15T20:41:15.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.429+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.434 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.429+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3742921142}) 2024-09-15T20:41:15.434 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.429+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1223633055}) 2024-09-15T20:41:15.435 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.435 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748823848}) 2024-09-15T20:41:15.436 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2160713381}) 2024-09-15T20:41:15.436 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.436 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=748823848}) 2024-09-15T20:41:15.436 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3219271935}) 2024-09-15T20:41:15.438 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.438 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821435727}) 2024-09-15T20:41:15.438 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3793190393}) 2024-09-15T20:41:15.438 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.438 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=821435727}) 2024-09-15T20:41:15.438 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.433+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=863497406}) 2024-09-15T20:41:15.440 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.440 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582219451}) 2024-09-15T20:41:15.440 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2602570097}) 2024-09-15T20:41:15.440 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.440 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3582219451}) 2024-09-15T20:41:15.440 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2875627286}) 2024-09-15T20:41:15.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483163396}) 2024-09-15T20:41:15.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3599417730}) 2024-09-15T20:41:15.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2483163396}) 2024-09-15T20:41:15.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.437+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3327839675}) 2024-09-15T20:41:15.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531432686}) 2024-09-15T20:41:15.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1343933105}) 2024-09-15T20:41:15.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.445 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=531432686}) 2024-09-15T20:41:15.445 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=605775523}) 2024-09-15T20:41:15.446 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.446 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=95,mdsmap=2,mgr=3} crc {logm=3622516324,mdsmap=215940802,mgr=1516913497}) 2024-09-15T20:41:15.446 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=1940169413,mdsmap=1235320281}) 2024-09-15T20:41:15.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=95,mdsmap=2,mgr=3} crc {logm=3622516324,mdsmap=215940802,mgr=1516913497}) 2024-09-15T20:41:15.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.441+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=98,mdsmap=2} crc {logm=2322519913,mdsmap=215940802}) 2024-09-15T20:41:15.449 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.445+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.449 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.445+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=25} crc {mgr=2831971963,mgr_command_descs=2551042220,mgr_metadata=1997488951,mgrstat=442972737,mon_config_key=2924772541,monmap=3785329752,nvmeofgw=3517310312,osd_metadata=2059034847,osd_pg_creating=123171471,osd_snap=2415305187,osdmap=943047635}) 2024-09-15T20:41:15.449 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.445+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=14,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=25} crc {mgr=2831971963,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=442972737,mon_config_key=46737265,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=123171471,osd_snap=3442893090,osdmap=3561646380}) 2024-09-15T20:41:15.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.445+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.445+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgr=5,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=15,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=30,osdmap=25} crc {mgr=2831971963,mgr_command_descs=2551042220,mgr_metadata=1997488951,mgrstat=442972737,mon_config_key=2924772541,monmap=3785329752,nvmeofgw=3517310312,osd_metadata=2059034847,osd_pg_creating=123171471,osd_snap=2415305187,osdmap=943047635}) 2024-09-15T20:41:15.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.445+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgr=4,mgr_command_descs=1,mgrstat=10,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=31,osdmap=25} crc {mgr=3256967926,mgr_command_descs=2551042219,mgrstat=442972737,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1681388020,osd_pg_creating=123171471,osd_snap=1663638911,osdmap=2373099327}) 2024-09-15T20:41:15.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.449+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.449+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1634723487}) 2024-09-15T20:41:15.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.449+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2057168607}) 2024-09-15T20:41:15.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.449+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.449+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1634723487}) 2024-09-15T20:41:15.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.449+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3387216452}) 2024-09-15T20:41:15.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.453+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.453+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3305719954}) 2024-09-15T20:41:15.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.453+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3460451648}) 2024-09-15T20:41:15.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.453+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:41:15.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.453+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3305719954}) 2024-09-15T20:41:15.456 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:41:15.453+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2224782127}) 2024-09-15T20:41:15.811 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-09-15T20:41:15.808+0000 7f1962dd1640 -1 snap_mapper.add_oid found existing snaps mapped on 300:fe4126ca:test-rados-api-smithi146-28320-80::foo:2, removing 2024-09-15T20:41:15.811 INFO:tasks.ceph.osd.0.smithi078.stderr:2024-09-15T20:41:15.805+0000 7ff77f6f6640 -1 snap_mapper.add_oid found existing snaps mapped on 300:fe4126ca:test-rados-api-smithi146-28320-80::foo:2, removing 2024-09-15T20:41:16.385 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:41:16.385 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.509+0000 7f612117f640 10 monclient: get_monmap_and_config 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.509+0000 7f612117f640 10 monclient: build_initial_monmap 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.509+0000 7f612117f640 10 monclient: monmap: 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:16.515415+0000 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:16.515415+0000 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:16.514 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:16.515 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:16.515 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.509+0000 7f612117f640 1 Processor -- start 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.509+0000 7f612117f640 1 -- start start 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.509+0000 7f612117f640 10 monclient: init 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient(hunting): picked mon.noname-b con 0x7f611c134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient(hunting): picked mon.noname-c con 0x7f611c09f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient(hunting): picked mon.noname-a con 0x7f611c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611c0a0470 con 0x7f611c06e960 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611c09feb0 con 0x7f611c09f5a0 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611c09e4a0 con 0x7f611c134bb0 2024-09-15T20:41:16.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient(hunting): authenticate will time out at 1737.400260s 2024-09-15T20:41:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f611ad9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f611c134bb0 0x7f611c134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35430/0 (socket says 172.21.15.78:35430) 2024-09-15T20:41:16.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f611ad9d640 1 -- 172.21.15.78:0/2510284309 learned_addr learned my addr 172.21.15.78:0/2510284309 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 102253974 0 0) 0x7f6114001120 con 0x7f611c06e960 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient(hunting): my global_id is 114129 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6104003530 con 0x7f611c06e960 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 869217096 0 0) 0x7f611c09feb0 con 0x7f611c09f5a0 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient(hunting): my global_id is 23623 2024-09-15T20:41:16.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61040048f0 con 0x7f611c09f5a0 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1293234174 0 0) 0x7f6104003530 con 0x7f611c06e960 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 >> v1:172.21.15.78:6790/0 conn(0x7f611c134bb0 legacy=0x7f611c134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 >> v1:172.21.15.146:6789/0 conn(0x7f611c09f5a0 legacy=0x7f611c0a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: _finish_hunting 0 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 monclient: found mon.noname-a 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611c0a2440 con 0x7f611c06e960 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: _finish_auth 0 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: _check_auth_tickets 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6114001850 con 0x7f611c06e960 2024-09-15T20:41:16.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 5 monclient: authenticate success, global_id 114129 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: dump: 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.520 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.521 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:16.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f61140019f0 con 0x7f611c06e960 2024-09-15T20:41:16.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:16.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:16.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 1 -- 172.21.15.78:0/2510284309 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6114003620 con 0x7f611c06e960 2024-09-15T20:41:16.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:16.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 4 set_mon_vals no callback set 2024-09-15T20:41:16.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f6119d9b640 10 monclient: dump: 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 10 monclient: shutdown 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:16.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 1 -- 172.21.15.78:0/2510284309 >> v1:172.21.15.78:6789/0 conn(0x7f611c06e960 legacy=0x7f611c0a1d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 1 -- 172.21.15.78:0/2510284309 shutdown_connections 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 1 -- 172.21.15.78:0/2510284309 >> 172.21.15.78:0/2510284309 conn(0x7f611c0a63c0 msgr2=0x7f611c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 1 -- 172.21.15.78:0/2510284309 shutdown_connections 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.513+0000 7f612117f640 1 -- 172.21.15.78:0/2510284309 wait complete. 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: build_initial_monmap 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: monmap: 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:16.527 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 librados: starting msgr at 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 librados: starting objecter 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 Processor -- start 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 -- start start 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 librados: setting wanted keys 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 librados: calling monclient init 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: init 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient(hunting): picked mon.noname-a con 0x7f611c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:16.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient(hunting): picked mon.noname-b con 0x7f611c09f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient(hunting): picked mon.noname-c con 0x7f611c134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611c13b0f0 con 0x7f611c06e960 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611c09d180 con 0x7f611c09f5a0 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f611c1403f0 con 0x7f611c134bb0 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient(hunting): authenticate will time out at 1737.405355s 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f611ad9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f611c06e960 0x7f611c13e640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42372/0 (socket says 172.21.15.78:42372) 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f611b59e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f611c134bb0 0x7f611c1d56d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35442/0 (socket says 172.21.15.78:35442) 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f611ad9d640 1 -- 172.21.15.78:0/3798901837 learned_addr learned my addr 172.21.15.78:0/3798901837 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 client.?.objecter ms_handle_connect 0x7f611c06e960 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 client.?.objecter ms_handle_connect 0x7f611c134bb0 2024-09-15T20:41:16.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007862226 0 0) 0x7f611c13b0f0 con 0x7f611c06e960 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): my global_id is 114132 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6108004ec0 con 0x7f611c06e960 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3339155490 0 0) 0x7f611c1403f0 con 0x7f611c134bb0 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): my global_id is 84575 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f611c13b0f0 con 0x7f611c134bb0 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 client.?.objecter ms_handle_connect 0x7f611c09f5a0 2024-09-15T20:41:16.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 290176877 0 0) 0x7f6108004ec0 con 0x7f611c06e960 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f611c1403f0 con 0x7f611c06e960 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f610c001cc0 con 0x7f611c06e960 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): dump: 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.531 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653719011 0 0) 0x7f611c09d180 con 0x7f611c09f5a0 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): my global_id is 23626 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6108004ec0 con 0x7f611c09f5a0 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1243099 0 0) 0x7f611c13b0f0 con 0x7f611c134bb0 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f611c09d180 con 0x7f611c134bb0 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6114001ba0 con 0x7f611c134bb0 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:16.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient(hunting): dump: 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 463198350 0 0) 0x7f611c09d180 con 0x7f611c134bb0 2024-09-15T20:41:16.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 >> v1:172.21.15.146:6789/0 conn(0x7f611c09f5a0 legacy=0x7f611c1d1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 >> v1:172.21.15.78:6789/0 conn(0x7f611c06e960 legacy=0x7f611c13e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient: _finish_hunting 0 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 monclient: found mon.c 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f611c13ae10 con 0x7f611c134bb0 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient: _finish_auth 0 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient: _check_auth_tickets 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 5 monclient: authenticate success, global_id 84575 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: _renew_subs 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f611c1398f0 con 0x7f611c134bb0 2024-09-15T20:41:16.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 client.84575.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: _renew_subs 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f611c13ef20 con 0x7f611c134bb0 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 librados: init done 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6114001d40 con 0x7f611c134bb0 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6114003c10 con 0x7f611c134bb0 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f611959a640 4 set_mon_vals no callback set 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 monclient: dump: 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.535 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f60fc005090 con 0x7f611c134bb0 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f6114003db0 con 0x7f611c134bb0 2024-09-15T20:41:16.536 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7f611404b5b0 con 0x7f611c134bb0 2024-09-15T20:41:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 10 client.84575.objecter ms_dispatch 0x7f611c000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 3 client.84575.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.517+0000 7f61037fe640 3 client.84575.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.521+0000 7f61037fe640 20 client.84575.objecter dump_active .. 0 homeless 2024-09-15T20:41:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.521+0000 7f611ad9d640 10 monclient: get_auth_request con 0x7f610803e240 auth_method 0 2024-09-15T20:41:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.521+0000 7f61037fe640 10 client.84575.objecter ms_handle_connect 0x7f610803e240 2024-09-15T20:41:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.525+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f611c0a0470 con 0x7f611c134bb0 2024-09-15T20:41:16.537 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.525+0000 7f61037fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:16.538 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.525+0000 7f61037fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.757+0000 7f612117f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:41:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.757+0000 7f612117f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.757+0000 7f612117f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:16.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.757+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f60fc005380 con 0x7f611c134bb0 2024-09-15T20:41:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f61037fe640 1 -- 172.21.15.78:0/3798901837 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 942382174) 0x7f6114016a90 con 0x7f611c134bb0 2024-09-15T20:41:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f61037fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:16.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f61037fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:16.764 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:41:16.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 10 librados: watch_flush enter 2024-09-15T20:41:16.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 10 librados: watch_flush exit 2024-09-15T20:41:16.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 20 client.84575.objecter shutdown clearing up homeless session... 2024-09-15T20:41:16.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 10 client.84575.objecter successfully canceled tick 2024-09-15T20:41:16.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 >> v1:172.21.15.146:6812/435598642 conn(0x7f610803e240 legacy=0x7f6108040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 10 monclient: shutdown 2024-09-15T20:41:16.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:16.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 >> v1:172.21.15.78:6790/0 conn(0x7f611c134bb0 legacy=0x7f611c1d56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 shutdown_connections 2024-09-15T20:41:16.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 >> 172.21.15.78:0/3798901837 conn(0x7f611c0a63c0 msgr2=0x7f611c0a4410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:16.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 shutdown_connections 2024-09-15T20:41:16.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 1 -- 172.21.15.78:0/3798901837 wait complete. 2024-09-15T20:41:16.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.761+0000 7f612117f640 1 librados: shutdown 2024-09-15T20:41:16.782 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:41:16.782 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:41:16.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:41:16.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:41:16.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient: get_monmap_and_config 2024-09-15T20:41:16.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient: build_initial_monmap 2024-09-15T20:41:16.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient: monmap: 2024-09-15T20:41:16.905 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:16.905 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.905 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:16.906501+0000 2024-09-15T20:41:16.905 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:16.906501+0000 2024-09-15T20:41:16.906 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:16.906 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.906 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:16.906 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:16.906 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:16.906 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 1 Processor -- start 2024-09-15T20:41:16.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 1 -- start start 2024-09-15T20:41:16.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient: init 2024-09-15T20:41:16.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:16.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:16.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient(hunting): picked mon.noname-c con 0x7fcdb4135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:16.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient(hunting): picked mon.noname-b con 0x7fcdb40a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient(hunting): picked mon.noname-a con 0x7fcdb413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb40745e0 con 0x7fcdb413c220 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb406ecf0 con 0x7fcdb4135b60 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb406efd0 con 0x7fcdb40a24e0 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 10 monclient(hunting): authenticate will time out at 1737.791073s 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcdb40a24e0 0x7fcdb413ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35450/0 (socket says 172.21.15.78:35450) 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb2ffd640 1 -- 172.21.15.78:0/3352163097 learned_addr learned my addr 172.21.15.78:0/3352163097 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1704408849 0 0) 0x7fcdb406efd0 con 0x7fcdb40a24e0 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient(hunting): my global_id is 84578 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd9c003530 con 0x7fcdb40a24e0 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3300202325 0 0) 0x7fcdb40745e0 con 0x7fcdb413c220 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient(hunting): my global_id is 114135 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdb406efd0 con 0x7fcdb413c220 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2257554568 0 0) 0x7fcdb406efd0 con 0x7fcdb413c220 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 >> v1:172.21.15.78:6790/0 conn(0x7fcdb40a24e0 legacy=0x7fcdb413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 >> v1:172.21.15.146:6789/0 conn(0x7fcdb4135b60 legacy=0x7fcdb4135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient: _finish_hunting 0 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 monclient: found mon.noname-a 2024-09-15T20:41:16.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb4136650 con 0x7fcdb413c220 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient: _finish_auth 0 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient: _check_auth_tickets 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcdac001930 con 0x7fcdb413c220 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 5 monclient: authenticate success, global_id 114135 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdb27fc640 10 monclient: dump: 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.916 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.901+0000 7fcdba124640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcdac001ad0 con 0x7fcdb413c220 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb27fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 4 set_mon_vals no callback set 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb27fc640 1 -- 172.21.15.78:0/3352163097 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcdac001ee0 con 0x7fcdb413c220 2024-09-15T20:41:16.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb27fc640 10 monclient: dump: 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: shutdown 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- 172.21.15.78:0/3352163097 >> v1:172.21.15.78:6789/0 conn(0x7fcdb413c220 legacy=0x7fcdb413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- 172.21.15.78:0/3352163097 shutdown_connections 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- 172.21.15.78:0/3352163097 >> 172.21.15.78:0/3352163097 conn(0x7fcdb40a63c0 msgr2=0x7fcdb40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- 172.21.15.78:0/3352163097 shutdown_connections 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- 172.21.15.78:0/3352163097 wait complete. 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: build_initial_monmap 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: monmap: 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:16.919 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 librados: starting msgr at 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 librados: starting objecter 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 Processor -- start 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- start start 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 librados: setting wanted keys 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 librados: calling monclient init 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: init 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:16.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient(hunting): picked mon.noname-b con 0x7fcdb40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient(hunting): picked mon.noname-c con 0x7fcdb4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient(hunting): picked mon.noname-a con 0x7fcdb413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb413f750 con 0x7fcdb413c220 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb409d180 con 0x7fcdb40a24e0 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcdb413a240 con 0x7fcdb4135b60 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient(hunting): authenticate will time out at 1737.794105s 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcdb4135b60 0x7fcdb409dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35454/0 (socket says 172.21.15.78:35454) 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb3fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcdb413c220 0x7fcdb409e2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42392/0 (socket says 172.21.15.78:42392) 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb2ffd640 1 -- 172.21.15.78:0/1914957470 learned_addr learned my addr 172.21.15.78:0/1914957470 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 client.?.objecter ms_handle_connect 0x7fcdb4135b60 2024-09-15T20:41:16.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 client.?.objecter ms_handle_connect 0x7fcdb413c220 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3663316360 0 0) 0x7fcdb413a240 con 0x7fcdb4135b60 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): my global_id is 84581 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcda0004ec0 con 0x7fcdb4135b60 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3847440688 0 0) 0x7fcdb4136650 con 0x7fcdb413c220 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): my global_id is 114138 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdb413a240 con 0x7fcdb413c220 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 client.?.objecter ms_handle_connect 0x7fcdb40a24e0 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2060736132 0 0) 0x7fcda0004ec0 con 0x7fcdb4135b60 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcdb4136650 con 0x7fcdb4135b60 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcda80013e0 con 0x7fcdb4135b60 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): dump: 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.923 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2062030920 0 0) 0x7fcdb413a240 con 0x7fcdb413c220 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcda0004ec0 con 0x7fcdb413c220 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcdac0017f0 con 0x7fcdb413c220 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient(hunting): dump: 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.924 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 800080665 0 0) 0x7fcdb4136650 con 0x7fcdb4135b60 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 >> v1:172.21.15.146:6789/0 conn(0x7fcdb40a24e0 legacy=0x7fcdb409d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 >> v1:172.21.15.78:6789/0 conn(0x7fcdb413c220 legacy=0x7fcdb409e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:41:16.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 monclient: found mon.c 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb409ee40 con 0x7fcdb4135b60 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient: _finish_auth 0 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 5 monclient: authenticate success, global_id 84581 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: _renew_subs 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:16.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb4139f60 con 0x7fcdb4135b60 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 client.84581.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: _renew_subs 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcdb41d5c40 con 0x7fcdb4135b60 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 librados: init done 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcda80015a0 con 0x7fcdb4135b60 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcda8004c40 con 0x7fcdb4135b60 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 10 monclient: dump: 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.927 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdb1ffb640 4 set_mon_vals no callback set 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcd940065d0 con 0x7fcd94002870 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.905+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fcda8010cb0 con 0x7fcdb4135b60 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 client.84581.objecter ms_handle_connect 0x7fcd94002870 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 client.84581.objecter resend_mon_ops 2024-09-15T20:41:16.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7fcda804b680 con 0x7fcdb4135b60 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 client.84581.objecter ms_dispatch 0x7fcdb4000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 3 client.84581.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 3 client.84581.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 20 client.84581.objecter dump_active .. 0 homeless 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4289719932 0 0) 0x7fcd940065d0 con 0x7fcd94002870 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient(hunting): my global_id is 114141 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcdb4136650 con 0x7fcd94002870 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3058520650 0 0) 0x7fcdb4136650 con 0x7fcd94002870 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd940065d0 con 0x7fcd94002870 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcda4001750 con 0x7fcd94002870 2024-09-15T20:41:16.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcdb3fff640 10 monclient: get_auth_request con 0x7fcda003e120 auth_method 0 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient: dump: 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3151848886 0 0) 0x7fcd940065d0 con 0x7fcd94002870 2024-09-15T20:41:16.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcd9400a170 con 0x7fcd94002870 2024-09-15T20:41:16.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 client.84581.objecter ms_handle_connect 0x7fcda003e120 2024-09-15T20:41:16.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fcda4001970 con 0x7fcd94002870 2024-09-15T20:41:16.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:16.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:16.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 >> v1:172.21.15.78:6789/0 conn(0x7fcd94002870 legacy=0x7fcd94004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.909+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 >> v1:172.21.15.78:6789/0 conn(0x7fcd94002870 legacy=0x7fcd94004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:16.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcdba124640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcd94009e60 con 0x7fcd94009120 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 client.84581.objecter ms_handle_connect 0x7fcd94009120 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 client.84581.objecter resend_mon_ops 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1364018363 0 0) 0x7fcd94009e60 con 0x7fcd94009120 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 monclient(hunting): my global_id is 114144 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd940065d0 con 0x7fcd94009120 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 245335509 0 0) 0x7fcd940065d0 con 0x7fcd94009120 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcd94009e60 con 0x7fcd94009120 2024-09-15T20:41:16.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcda40037c0 con 0x7fcd94009120 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.913+0000 7fcd9bfff640 10 monclient: dump: 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:16.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 372977608 0 0) 0x7fcd94009e60 con 0x7fcd94009120 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcd9400c400 con 0x7fcd94009120 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3971 (unknown 0 0 1558747881) 0x7fcda4003620 con 0x7fcd94009120 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcd9bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcd9bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 >> v1:172.21.15.78:6789/0 conn(0x7fcd94009120 legacy=0x7fcd940095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcd9bfff640 1 -- 172.21.15.78:0/1914957470 >> v1:172.21.15.78:6789/0 conn(0x7fcd94009120 legacy=0x7fcd940095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:16.935 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 38, 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 2759, 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:16.936 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:16.937 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:41:16.938 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:16.939 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:16.940 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:16.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:16.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:16.943 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:41:16.944 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:16.945 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:41:16.946 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:41:16.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 10 librados: watch_flush enter 2024-09-15T20:41:16.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 10 librados: watch_flush exit 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 20 client.84581.objecter shutdown clearing up homeless session... 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 10 client.84581.objecter successfully canceled tick 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 >> v1:172.21.15.146:6812/435598642 conn(0x7fcda003e120 legacy=0x7fcda00405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 10 monclient: shutdown 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 >> v1:172.21.15.78:6790/0 conn(0x7fcdb4135b60 legacy=0x7fcdb409dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdb3fff640 1 -- 172.21.15.78:0/1914957470 reap_dead start 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdb3fff640 1 -- 172.21.15.78:0/1914957470 reap_dead start 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 shutdown_connections 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 >> 172.21.15.78:0/1914957470 conn(0x7fcdb40a63c0 msgr2=0x7fcdb40a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 shutdown_connections 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 1 -- 172.21.15.78:0/1914957470 wait complete. 2024-09-15T20:41:16.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:16.917+0000 7fcdba124640 1 librados: shutdown 2024-09-15T20:41:16.949 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:41:16.949 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:41:16.949 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.053+0000 7f59b13fa640 10 monclient: get_monmap_and_config 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.053+0000 7f59b13fa640 10 monclient: build_initial_monmap 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.053+0000 7f59b13fa640 10 monclient: monmap: 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:17.060624+0000 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:17.060624+0000 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:17.060 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:17.061 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:17.061 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 Processor -- start 2024-09-15T20:41:17.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- start start 2024-09-15T20:41:17.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient: init 2024-09-15T20:41:17.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:17.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:41:17.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient(hunting): picked mon.noname-a con 0x7f59ac135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient(hunting): picked mon.noname-b con 0x7f59ac0a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient(hunting): picked mon.noname-c con 0x7f59ac13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ac0745e0 con 0x7f59ac135b60 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ac06ecf0 con 0x7f59ac13c220 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ac06efd0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient(hunting): authenticate will time out at 1737.945149s 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59aaffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f59ac135b60 0x7f59ac135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42404/0 (socket says 172.21.15.78:42404) 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59aa7fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f59ac0a24e0 0x7f59ac13ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35458/0 (socket says 172.21.15.78:35458) 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59aaffd640 1 -- 172.21.15.78:0/1937554560 learned_addr learned my addr 172.21.15.78:0/1937554560 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1435116234 0 0) 0x7f59ac0745e0 con 0x7f59ac135b60 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient(hunting): my global_id is 114147 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5994003530 con 0x7f59ac135b60 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3118304276 0 0) 0x7f59ac06efd0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient(hunting): my global_id is 84584 2024-09-15T20:41:17.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59ac0745e0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1208655110 0 0) 0x7f5994003530 con 0x7f59ac135b60 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 >> v1:172.21.15.78:6790/0 conn(0x7f59ac0a24e0 legacy=0x7f59ac13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 >> v1:172.21.15.146:6789/0 conn(0x7f59ac13c220 legacy=0x7f59ac13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 monclient: found mon.noname-a 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ac136650 con 0x7f59ac135b60 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f59a00016a0 con 0x7f59ac135b60 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 5 monclient: authenticate success, global_id 114147 2024-09-15T20:41:17.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: dump: 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59a00018c0 con 0x7f59ac135b60 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 4 set_mon_vals no callback set 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 1 -- 172.21.15.78:0/1937554560 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f59a0001c90 con 0x7f59ac135b60 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59a9ffb640 10 monclient: dump: 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.068 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient: shutdown 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- 172.21.15.78:0/1937554560 >> v1:172.21.15.78:6789/0 conn(0x7f59ac135b60 legacy=0x7f59ac135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- 172.21.15.78:0/1937554560 shutdown_connections 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- 172.21.15.78:0/1937554560 >> 172.21.15.78:0/1937554560 conn(0x7f59ac0a63c0 msgr2=0x7f59ac0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- 172.21.15.78:0/1937554560 shutdown_connections 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 -- 172.21.15.78:0/1937554560 wait complete. 2024-09-15T20:41:17.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient: build_initial_monmap 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 10 monclient: monmap: 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 librados: starting msgr at 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.057+0000 7f59b13fa640 1 librados: starting objecter 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 Processor -- start 2024-09-15T20:41:17.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 -- start start 2024-09-15T20:41:17.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 librados: setting wanted keys 2024-09-15T20:41:17.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 librados: calling monclient init 2024-09-15T20:41:17.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient: init 2024-09-15T20:41:17.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:17.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:41:17.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient(hunting): picked mon.noname-c con 0x7f59ac0a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:17.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient(hunting): picked mon.noname-a con 0x7f59ac135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient(hunting): picked mon.noname-b con 0x7f59ac13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ac09e2b0 con 0x7f59ac135b60 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ac09d180 con 0x7f59ac13c220 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59aaffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f59ac0a24e0 0x7f59ac09d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35468/0 (socket says 172.21.15.78:35468) 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59aaffd640 1 -- 172.21.15.78:0/622810179 learned_addr learned my addr 172.21.15.78:0/622810179 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59ac13bde0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient(hunting): authenticate will time out at 1737.948960s 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.?.objecter ms_handle_connect 0x7f59ac0a24e0 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:17.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.?.objecter ms_handle_connect 0x7f59ac135b60 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2107354604 0 0) 0x7f59ac13bde0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): my global_id is 84587 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5998004ec0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 291459337 0 0) 0x7f59ac09e2b0 con 0x7f59ac135b60 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): my global_id is 114150 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59ac13bde0 con 0x7f59ac135b60 2024-09-15T20:41:17.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.?.objecter ms_handle_connect 0x7f59ac13c220 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3507203988 0 0) 0x7f5998004ec0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f59ac09e2b0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f59a0001c10 con 0x7f59ac0a24e0 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): dump: 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.079 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2156250402 0 0) 0x7f59ac13bde0 con 0x7f59ac135b60 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5998004ec0 con 0x7f59ac135b60 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4101051667 0 0) 0x7f59ac09d180 con 0x7f59ac13c220 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): my global_id is 23638 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59ac13bde0 con 0x7f59ac13c220 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f599c0018c0 con 0x7f59ac135b60 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): dump: 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.081 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1909976441 0 0) 0x7f59ac09e2b0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 >> v1:172.21.15.146:6789/0 conn(0x7f59ac13c220 legacy=0x7f59ac1d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 >> v1:172.21.15.78:6789/0 conn(0x7f59ac135b60 legacy=0x7f59ac09dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: _finish_hunting 0 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 monclient: found mon.c 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59ac0723a0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: _finish_auth 0 2024-09-15T20:41:17.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: _check_auth_tickets 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 5 monclient: authenticate success, global_id 84587 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient: _renew_subs 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f59ac0713b0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 client.84587.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient: _renew_subs 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f59ac13b7c0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 librados: init done 2024-09-15T20:41:17.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f59a0001820 con 0x7f59ac0a24e0 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f59a0004da0 con 0x7f59ac0a24e0 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59a97fa640 4 set_mon_vals no callback set 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: dump: 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.084 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f598c0065d0 con 0x7f598c002870 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.84587.objecter ms_handle_connect 0x7f598c002870 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.84587.objecter resend_mon_ops 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f59a0010e10 con 0x7f59ac0a24e0 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7f59a004b770 con 0x7f59ac0a24e0 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.84587.objecter ms_dispatch 0x7f59ac000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:17.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 3 client.84587.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 3 client.84587.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 20 client.84587.objecter dump_active .. 0 homeless 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1676002640 0 0) 0x7f598c0065d0 con 0x7f598c002870 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient(hunting): my global_id is 84590 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f59ac09e2b0 con 0x7f598c002870 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59ab7fe640 10 monclient: get_auth_request con 0x7f599803e170 auth_method 0 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3208767563 0 0) 0x7f59ac09e2b0 con 0x7f598c002870 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f598c0065d0 con 0x7f598c002870 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f599c0047f0 con 0x7f598c002870 2024-09-15T20:41:17.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: dump: 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1681056186 0 0) 0x7f598c0065d0 con 0x7f598c002870 2024-09-15T20:41:17.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f598c00a170 con 0x7f598c002870 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 client.84587.objecter ms_handle_connect 0x7f599803e170 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f599c005820 con 0x7f598c002870 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 >> v1:172.21.15.78:6790/0 conn(0x7f598c002870 legacy=0x7f598c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.061+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 >> v1:172.21.15.78:6790/0 conn(0x7f598c002870 legacy=0x7f598c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f598c009e30 con 0x7f598c0090f0 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 client.84587.objecter ms_handle_connect 0x7f598c0090f0 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 client.84587.objecter resend_mon_ops 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4127459947 0 0) 0x7f598c009e30 con 0x7f598c0090f0 2024-09-15T20:41:17.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient(hunting): my global_id is 84593 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f598c0065d0 con 0x7f598c0090f0 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 804619349 0 0) 0x7f598c0065d0 con 0x7f598c0090f0 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f598c009e30 con 0x7f598c0090f0 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f599c004de0 con 0x7f598c0090f0 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient: dump: 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.089 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2831261137 0 0) 0x7f598c009e30 con 0x7f598c0090f0 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f598c00c400 con 0x7f598c0090f0 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 3623113687) 0x7f599c004e80 con 0x7f598c0090f0 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 >> v1:172.21.15.78:6790/0 conn(0x7f598c0090f0 legacy=0x7f598c009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59937fe640 1 -- 172.21.15.78:0/622810179 >> v1:172.21.15.78:6790/0 conn(0x7f598c0090f0 legacy=0x7f598c009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 38, 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 348756, 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:41:17.091 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:17.092 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:17.093 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:41:17.094 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:17.095 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.096 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:41:17.097 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:41:17.098 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:17.099 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:17.100 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:17.101 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:41:17.102 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:41:17.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 10 librados: watch_flush enter 2024-09-15T20:41:17.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 10 librados: watch_flush exit 2024-09-15T20:41:17.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 20 client.84587.objecter shutdown clearing up homeless session... 2024-09-15T20:41:17.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 10 client.84587.objecter successfully canceled tick 2024-09-15T20:41:17.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 >> v1:172.21.15.146:6812/435598642 conn(0x7f599803e170 legacy=0x7f5998040630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 10 monclient: shutdown 2024-09-15T20:41:17.104 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:17.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 >> v1:172.21.15.78:6790/0 conn(0x7f59ac0a24e0 legacy=0x7f59ac09d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59ab7fe640 1 -- 172.21.15.78:0/622810179 reap_dead start 2024-09-15T20:41:17.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59ab7fe640 1 -- 172.21.15.78:0/622810179 reap_dead start 2024-09-15T20:41:17.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 shutdown_connections 2024-09-15T20:41:17.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 >> 172.21.15.78:0/622810179 conn(0x7f59ac0a63c0 msgr2=0x7f59ac0a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:17.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 shutdown_connections 2024-09-15T20:41:17.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 1 -- 172.21.15.78:0/622810179 wait complete. 2024-09-15T20:41:17.105 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.069+0000 7f59b13fa640 1 librados: shutdown 2024-09-15T20:41:17.105 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:41:17.105 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:41:17.106 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:41:17.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.213+0000 7f5ab1ff2640 10 monclient: get_monmap_and_config 2024-09-15T20:41:17.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.213+0000 7f5ab1ff2640 10 monclient: build_initial_monmap 2024-09-15T20:41:17.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.213+0000 7f5ab1ff2640 10 monclient: monmap: 2024-09-15T20:41:17.219 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:17.219 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:17.220813+0000 2024-09-15T20:41:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:17.220813+0000 2024-09-15T20:41:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:17.220 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:17.220 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 Processor -- start 2024-09-15T20:41:17.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- start start 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient: init 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient(hunting): picked mon.noname-b con 0x7f5aac134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient(hunting): picked mon.noname-c con 0x7f5aac09f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient(hunting): picked mon.noname-a con 0x7f5aac06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac0a0470 con 0x7f5aac06e960 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac09feb0 con 0x7f5aac09f5a0 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac09e4a0 con 0x7f5aac134bb0 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient(hunting): authenticate will time out at 1738.105132s 2024-09-15T20:41:17.228 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aab7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f5aac134bb0 0x7f5aac134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35488/0 (socket says 172.21.15.78:35488) 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aab7fe640 1 -- 172.21.15.78:0/3827710445 learned_addr learned my addr 172.21.15.78:0/3827710445 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1171698151 0 0) 0x7f5aa4001120 con 0x7f5aac06e960 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient(hunting): my global_id is 114153 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a94003530 con 0x7f5aac06e960 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3293928917 0 0) 0x7f5aac09e4a0 con 0x7f5aac134bb0 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient(hunting): my global_id is 84596 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aa4001120 con 0x7f5aac134bb0 2024-09-15T20:41:17.229 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1890929947 0 0) 0x7f5aa4001120 con 0x7f5aac134bb0 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 >> v1:172.21.15.146:6789/0 conn(0x7f5aac09f5a0 legacy=0x7f5aac1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 >> v1:172.21.15.78:6789/0 conn(0x7f5aac06e960 legacy=0x7f5aac13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 monclient: found mon.noname-b 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac0a0ff0 con 0x7f5aac134bb0 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: _finish_auth 0 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 5 monclient: authenticate success, global_id 84596 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5a9c001a00 con 0x7f5aac134bb0 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.230 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: dump: 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5a9c001c20 con 0x7f5aac134bb0 2024-09-15T20:41:17.231 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 1 -- 172.21.15.78:0/3827710445 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5a9c002030 con 0x7f5aac134bb0 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 4 set_mon_vals no callback set 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5aaa7fc640 10 monclient: dump: 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.232 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient: shutdown 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/3827710445 >> v1:172.21.15.78:6790/0 conn(0x7f5aac134bb0 legacy=0x7f5aac134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/3827710445 shutdown_connections 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/3827710445 >> 172.21.15.78:0/3827710445 conn(0x7f5aac0a63c0 msgr2=0x7f5aac0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/3827710445 shutdown_connections 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/3827710445 wait complete. 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient: build_initial_monmap 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient: monmap: 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:17.233 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 librados: starting msgr at 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 librados: starting objecter 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 Processor -- start 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 -- start start 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 librados: setting wanted keys 2024-09-15T20:41:17.234 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 1 librados: calling monclient init 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.217+0000 7f5ab1ff2640 10 monclient: init 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient(hunting): picked mon.noname-b con 0x7f5aac06e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient(hunting): picked mon.noname-c con 0x7f5aac09f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient(hunting): picked mon.noname-a con 0x7f5aac134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac13b9f0 con 0x7f5aac134bb0 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac13cad0 con 0x7f5aac06e960 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5aac13cdb0 con 0x7f5aac09f5a0 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient(hunting): authenticate will time out at 1738.108166s 2024-09-15T20:41:17.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5aaaffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f5aac09f5a0 0x7f5aac1d1b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35492/0 (socket says 172.21.15.78:35492) 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5aaaffd640 1 -- 172.21.15.78:0/4286925294 learned_addr learned my addr 172.21.15.78:0/4286925294 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.?.objecter ms_handle_connect 0x7f5aac09f5a0 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.?.objecter ms_handle_connect 0x7f5aac134bb0 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1135129913 0 0) 0x7f5aac13cdb0 con 0x7f5aac09f5a0 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): my global_id is 84599 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a98004ec0 con 0x7f5aac09f5a0 2024-09-15T20:41:17.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1461869403 0 0) 0x7f5aac13b9f0 con 0x7f5aac134bb0 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): my global_id is 114156 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aac13cdb0 con 0x7f5aac134bb0 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.?.objecter ms_handle_connect 0x7f5aac06e960 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 348830819 0 0) 0x7f5aac13cdb0 con 0x7f5aac134bb0 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5aac13b9f0 con 0x7f5aac134bb0 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5aa4001630 con 0x7f5aac134bb0 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:17.237 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): dump: 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3688033263 0 0) 0x7f5a98004ec0 con 0x7f5aac09f5a0 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5aac13cdb0 con 0x7f5aac09f5a0 2024-09-15T20:41:17.238 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5aa0001990 con 0x7f5aac09f5a0 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): dump: 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1127034873 0 0) 0x7f5aac13cad0 con 0x7f5aac06e960 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): my global_id is 23644 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a98004ec0 con 0x7f5aac06e960 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1730925962 0 0) 0x7f5aac13b9f0 con 0x7f5aac134bb0 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 >> v1:172.21.15.78:6790/0 conn(0x7f5aac09f5a0 legacy=0x7f5aac1d1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 >> v1:172.21.15.146:6789/0 conn(0x7f5aac06e960 legacy=0x7f5aac133f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: _finish_hunting 0 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 monclient: found mon.a 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5aac0a45e0 con 0x7f5aac134bb0 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: _finish_auth 0 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: _check_auth_tickets 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:17.240 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 5 monclient: authenticate success, global_id 114156 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient: _renew_subs 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5aac09d180 con 0x7f5aac134bb0 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 client.114156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient: _renew_subs 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5aac13b2e0 con 0x7f5aac134bb0 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 1 librados: init done 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5aa4003600 con 0x7f5aac134bb0 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5aa4003950 con 0x7f5aac134bb0 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:17.241 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5aa9ffb640 4 set_mon_vals no callback set 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: dump: 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5a8c0065d0 con 0x7f5a8c002870 2024-09-15T20:41:17.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f5aa400f9c0 con 0x7f5aac134bb0 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.114156.objecter ms_handle_connect 0x7f5a8c002870 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.114156.objecter resend_mon_ops 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7f5aa404a330 con 0x7f5aac134bb0 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.114156.objecter ms_dispatch 0x7f5aac000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 3 client.114156.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 3 client.114156.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 20 client.114156.objecter dump_active .. 0 homeless 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2859146371 0 0) 0x7f5a8c0065d0 con 0x7f5a8c002870 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient(hunting): my global_id is 23647 2024-09-15T20:41:17.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5aac13b9f0 con 0x7f5a8c002870 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5aaaffd640 10 monclient: get_auth_request con 0x7f5a9803e220 auth_method 0 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2484729054 0 0) 0x7f5aac13b9f0 con 0x7f5a8c002870 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a8c0065d0 con 0x7f5a8c002870 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5a9c001a80 con 0x7f5a8c002870 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 monclient: dump: 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.244 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 10 client.114156.objecter ms_handle_connect 0x7f5a9803e220 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1006881591 0 0) 0x7f5a8c0065d0 con 0x7f5a8c002870 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.221+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5a8c00a170 con 0x7f5a8c002870 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.225+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5a9c001de0 con 0x7f5a8c002870 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.225+0000 7f5a93fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.225+0000 7f5a93fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.225+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 >> v1:172.21.15.146:6789/0 conn(0x7f5a8c002870 legacy=0x7f5a8c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.245 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.225+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 >> v1:172.21.15.146:6789/0 conn(0x7f5a8c002870 legacy=0x7f5a8c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5ab1ff2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:41:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5a8c009e80 con 0x7f5a8c009140 2024-09-15T20:41:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 client.114156.objecter ms_handle_connect 0x7f5a8c009140 2024-09-15T20:41:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 client.114156.objecter resend_mon_ops 2024-09-15T20:41:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 612121530 0 0) 0x7f5a8c009e80 con 0x7f5a8c009140 2024-09-15T20:41:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 monclient(hunting): my global_id is 23650 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a8c0065d0 con 0x7f5a8c009140 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 623643453 0 0) 0x7f5a8c0065d0 con 0x7f5a8c009140 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5a8c009e80 con 0x7f5a8c009140 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5a9c0051a0 con 0x7f5a8c009140 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 10 monclient: dump: 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.248 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2313627621 0 0) 0x7f5a8c009e80 con 0x7f5a8c009140 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.229+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5a8c00c420 con 0x7f5a8c009140 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 3166843180) 0x7f5a9c005130 con 0x7f5a8c009140 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5a93fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:41:17.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5a93fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 >> v1:172.21.15.146:6789/0 conn(0x7f5a8c009140 legacy=0x7f5a8c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5a93fff640 1 -- 172.21.15.78:0/4286925294 >> v1:172.21.15.146:6789/0 conn(0x7f5a8c009140 legacy=0x7f5a8c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 38, 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 589741, 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:41:17.250 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:17.251 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:17.252 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:17.252 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:17.252 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:17.253 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:41:17.254 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:41:17.255 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:17.256 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.257 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.258 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.259 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:17.260 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.261 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:17.262 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.263 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:17.264 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:17.265 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:17.265 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:17.265 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:17.265 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:17.265 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:41:17.265 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:41:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 10 librados: watch_flush enter 2024-09-15T20:41:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 10 librados: watch_flush exit 2024-09-15T20:41:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 20 client.114156.objecter shutdown clearing up homeless session... 2024-09-15T20:41:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 10 client.114156.objecter successfully canceled tick 2024-09-15T20:41:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 >> v1:172.21.15.146:6812/435598642 conn(0x7f5a9803e220 legacy=0x7f5a980406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 10 monclient: shutdown 2024-09-15T20:41:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:17.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 >> v1:172.21.15.78:6789/0 conn(0x7f5aac134bb0 legacy=0x7f5aac1d52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5aabfff640 1 -- 172.21.15.78:0/4286925294 reap_dead start 2024-09-15T20:41:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5aabfff640 1 -- 172.21.15.78:0/4286925294 reap_dead start 2024-09-15T20:41:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 shutdown_connections 2024-09-15T20:41:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 >> 172.21.15.78:0/4286925294 conn(0x7f5aac0a63c0 msgr2=0x7f5aac0a1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 shutdown_connections 2024-09-15T20:41:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 1 -- 172.21.15.78:0/4286925294 wait complete. 2024-09-15T20:41:17.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.233+0000 7f5ab1ff2640 1 librados: shutdown 2024-09-15T20:41:17.268 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:41:17.269 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T20:41:17.269 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T20:41:17.269 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T20:41:17.269 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T20:41:17.269 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T20:41:17.269 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:41:17.282 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T20:41:17.282 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:41:17.282 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:41:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient: get_monmap_and_config 2024-09-15T20:41:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient: build_initial_monmap 2024-09-15T20:41:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient: monmap: 2024-09-15T20:41:17.433 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:17.434778+0000 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:17.434778+0000 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:17.434 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 1 Processor -- start 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 1 -- start start 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient: init 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient(hunting): picked mon.noname-c con 0x7f0570074630 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient(hunting): picked mon.noname-b con 0x7f057009d950 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient(hunting): picked mon.noname-a con 0x7f0570074a10 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0570134bc0 con 0x7f0570074a10 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057009f900 con 0x7f0570074630 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057009fbe0 con 0x7f057009d950 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:17.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 10 monclient(hunting): authenticate will time out at 1738.319183s 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f05677fe640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f057009d950 legacy=0x7f05701344b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0574cf1640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0570074a10 0x7f057013d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42458/0 (socket says 172.21.15.78:42458) 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0574cf1640 1 -- 172.21.15.78:0/4151010906 learned_addr learned my addr 172.21.15.78:0/4151010906 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2560587963 0 0) 0x7f0570134bc0 con 0x7f0570074a10 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient(hunting): my global_id is 114159 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f055c003530 con 0x7f0570074a10 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4118145221 0 0) 0x7f055c003530 con 0x7f0570074a10 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 >> v1:172.21.15.78:6790/0 conn(0x7f057009d950 legacy=0x7f05701344b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 >> v1:172.21.15.146:6789/0 conn(0x7f0570074630 legacy=0x7f0570133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 1 monclient: found mon.noname-a 2024-09-15T20:41:17.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057009fec0 con 0x7f0570074a10 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 5 monclient: authenticate success, global_id 114159 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0568001a20 con 0x7f0570074a10 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f056f7fe640 10 monclient: dump: 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.442 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.429+0000 7f0576754640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0568001c40 con 0x7f0570074a10 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056f7fe640 1 -- 172.21.15.78:0/4151010906 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0568002050 con 0x7f0570074a10 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056f7fe640 10 monclient: dump: 2024-09-15T20:41:17.443 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 4 set_mon_vals no callback set 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: shutdown 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- 172.21.15.78:0/4151010906 >> v1:172.21.15.78:6789/0 conn(0x7f0570074a10 legacy=0x7f057013d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- 172.21.15.78:0/4151010906 shutdown_connections 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- 172.21.15.78:0/4151010906 >> 172.21.15.78:0/4151010906 conn(0x7f05700a63c0 msgr2=0x7f05700a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- 172.21.15.78:0/4151010906 shutdown_connections 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- 172.21.15.78:0/4151010906 wait complete. 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: build_initial_monmap 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: monmap: 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 librados: starting msgr at 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 librados: starting objecter 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 Processor -- start 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- start start 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 librados: setting wanted keys 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 librados: calling monclient init 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: init 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient(hunting): picked mon.noname-b con 0x7f0570074630 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient(hunting): picked mon.noname-c con 0x7f0570074a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient(hunting): picked mon.noname-a con 0x7f057009d950 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0570071df0 con 0x7f057009d950 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057009d180 con 0x7f0570074630 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0570074150 con 0x7f0570074a10 2024-09-15T20:41:17.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient(hunting): authenticate will time out at 1738.322193s 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f05677fe640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f0570074a10 legacy=0x7f05701cf6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0574cf1640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f057009d950 0x7f05701d2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:42468/0 (socket says 172.21.15.78:42468) 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0574cf1640 1 -- 172.21.15.78:0/3707279839 learned_addr learned my addr 172.21.15.78:0/3707279839 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 client.?.objecter ms_handle_connect 0x7f057009d950 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 440116646 0 0) 0x7f0570071df0 con 0x7f057009d950 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient(hunting): my global_id is 114162 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0550004ef0 con 0x7f057009d950 2024-09-15T20:41:17.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 client.?.objecter ms_handle_connect 0x7f0570074630 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3378353135 0 0) 0x7f0550004ef0 con 0x7f057009d950 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0570071df0 con 0x7f057009d950 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0568001f90 con 0x7f057009d950 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient(hunting): dump: 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.448 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1066792330 0 0) 0x7f0570071df0 con 0x7f057009d950 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 >> v1:172.21.15.78:6790/0 conn(0x7f0570074a10 legacy=0x7f05701cf6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 >> v1:172.21.15.146:6789/0 conn(0x7f0570074630 legacy=0x7f057012db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 monclient: found mon.a 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0570072e10 con 0x7f057009d950 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient: _finish_auth 0 2024-09-15T20:41:17.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 5 monclient: authenticate success, global_id 114162 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: _renew_subs 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f05700738b0 con 0x7f057009d950 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 client.114162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: _renew_subs 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f05701ca5f0 con 0x7f057009d950 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 librados: init done 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0568001ba0 con 0x7f057009d950 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0568005160 con 0x7f057009d950 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:17.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056d7fa640 10 monclient: dump: 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f056effd640 4 set_mon_vals no callback set 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:17.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.433+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0554005090 con 0x7f057009d950 2024-09-15T20:41:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.437+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f05680111d0 con 0x7f057009d950 2024-09-15T20:41:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.437+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7f056804ba10 con 0x7f057009d950 2024-09-15T20:41:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.437+0000 7f056d7fa640 10 client.114162.objecter ms_dispatch 0x7f0570000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.437+0000 7f056d7fa640 3 client.114162.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.437+0000 7f056d7fa640 3 client.114162.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.437+0000 7f056d7fa640 20 client.114162.objecter dump_active .. 0 homeless 2024-09-15T20:41:17.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.437+0000 7f056ffff640 10 monclient: get_auth_request con 0x7f055003df50 auth_method 0 2024-09-15T20:41:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.437+0000 7f056d7fa640 10 client.114162.objecter ms_handle_connect 0x7f055003df50 2024-09-15T20:41:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.441+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f056801b210 con 0x7f057009d950 2024-09-15T20:41:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.441+0000 7f056d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:17.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.441+0000 7f056d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.665+0000 7f0576754640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:41:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.665+0000 7f0576754640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.665+0000 7f0576754640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.665+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0554005380 con 0x7f057009d950 2024-09-15T20:41:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.665+0000 7f056d7fa640 1 -- 172.21.15.78:0/3707279839 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3305746012) 0x7f0568020150 con 0x7f057009d950 2024-09-15T20:41:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.665+0000 7f056d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:17.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.665+0000 7f056d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:17.672 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:41:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 10 librados: watch_flush enter 2024-09-15T20:41:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 10 librados: watch_flush exit 2024-09-15T20:41:17.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 20 client.114162.objecter shutdown clearing up homeless session... 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 10 client.114162.objecter successfully canceled tick 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 >> v1:172.21.15.146:6812/435598642 conn(0x7f055003df50 legacy=0x7f0550040410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 10 monclient: shutdown 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 >> v1:172.21.15.78:6789/0 conn(0x7f057009d950 legacy=0x7f05701d2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 shutdown_connections 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 >> 172.21.15.78:0/3707279839 conn(0x7f05700a63c0 msgr2=0x7f05700a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 shutdown_connections 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 1 -- 172.21.15.78:0/3707279839 wait complete. 2024-09-15T20:41:17.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:17.669+0000 7f0576754640 1 librados: shutdown 2024-09-15T20:41:20.689 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:41:20.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient: get_monmap_and_config 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient: build_initial_monmap 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient: monmap: 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:20.822180+0000 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:20.822180+0000 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:20.821 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 1 Processor -- start 2024-09-15T20:41:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 1 -- start start 2024-09-15T20:41:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient: init 2024-09-15T20:41:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:41:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient(hunting): picked mon.noname-b con 0x7f1ffc135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:20.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient(hunting): picked mon.noname-c con 0x7f1ffc0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient(hunting): picked mon.noname-a con 0x7f1ffc13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc0745e0 con 0x7f1ffc13c220 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc06ecf0 con 0x7f1ffc0a24e0 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc06efd0 con 0x7f1ffc135b60 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f2001a31640 10 monclient(hunting): authenticate will time out at 1741.707265s 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ffaffd640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f1ffc135b60 legacy=0x7f1ffc135f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ffb7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1ffc13c220 0x7f1ffc13e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36632/0 (socket says 172.21.15.78:36632) 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ffb7fe640 1 -- 172.21.15.78:0/89392219 learned_addr learned my addr 172.21.15.78:0/89392219 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2874194796 0 0) 0x7f1ffc0745e0 con 0x7f1ffc13c220 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ff9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:20.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ff9ffb640 10 monclient(hunting): my global_id is 114165 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.817+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe4003530 con 0x7f1ffc13c220 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3323096098 0 0) 0x7f1fe4003530 con 0x7f1ffc13c220 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 >> v1:172.21.15.78:6790/0 conn(0x7f1ffc135b60 legacy=0x7f1ffc135f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 >> v1:172.21.15.146:6789/0 conn(0x7f1ffc0a24e0 legacy=0x7f1ffc13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 1 monclient: found mon.noname-a 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc136650 con 0x7f1ffc13c220 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:41:20.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1ff4001a20 con 0x7f1ffc13c220 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 5 monclient: authenticate success, global_id 114165 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: dump: 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:20.827 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ff4001c40 con 0x7f1ffc13c220 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 1 -- 172.21.15.78:0/89392219 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1ff4002050 con 0x7f1ffc13c220 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 4 set_mon_vals no callback set 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff9ffb640 10 monclient: dump: 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:20.828 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: shutdown 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- 172.21.15.78:0/89392219 >> v1:172.21.15.78:6789/0 conn(0x7f1ffc13c220 legacy=0x7f1ffc13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- 172.21.15.78:0/89392219 shutdown_connections 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- 172.21.15.78:0/89392219 >> 172.21.15.78:0/89392219 conn(0x7f1ffc0a63c0 msgr2=0x7f1ffc0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:20.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- 172.21.15.78:0/89392219 shutdown_connections 2024-09-15T20:41:20.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- 172.21.15.78:0/89392219 wait complete. 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: build_initial_monmap 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: monmap: 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:20.833 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 librados: starting msgr at 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 librados: starting objecter 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 Processor -- start 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- start start 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 librados: setting wanted keys 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 librados: calling monclient init 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: init 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:41:20.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient(hunting): picked mon.noname-b con 0x7f1ffc0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient(hunting): picked mon.noname-c con 0x7f1ffc135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient(hunting): picked mon.noname-a con 0x7f1ffc13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc137cb0 con 0x7f1ffc13c220 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc1cb9e0 con 0x7f1ffc0a24e0 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ffa7fc640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f1ffc135b60 legacy=0x7f1ffc1cb2d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ffc13fad0 con 0x7f1ffc135b60 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient(hunting): authenticate will time out at 1741.710545s 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ffb7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1ffc13c220 0x7f1ffc1d2e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36640/0 (socket says 172.21.15.78:36640) 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ffb7fe640 1 -- 172.21.15.78:0/4245127788 learned_addr learned my addr 172.21.15.78:0/4245127788 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 client.?.objecter ms_handle_connect 0x7f1ffc13c220 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:20.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 831934931 0 0) 0x7f1ffc137cb0 con 0x7f1ffc13c220 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient(hunting): _negotiate 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient(hunting): my global_id is 114168 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1fe8004ec0 con 0x7f1ffc13c220 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 client.?.objecter ms_handle_connect 0x7f1ffc0a24e0 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2966401819 0 0) 0x7f1fe8004ec0 con 0x7f1ffc13c220 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1ffc137cb0 con 0x7f1ffc13c220 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1ff4001f90 con 0x7f1ffc13c220 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:20.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient(hunting): dump: 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1919064560 0 0) 0x7f1ffc137cb0 con 0x7f1ffc13c220 2024-09-15T20:41:20.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 >> v1:172.21.15.78:6790/0 conn(0x7f1ffc135b60 legacy=0x7f1ffc1cb2d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 >> v1:172.21.15.146:6789/0 conn(0x7f1ffc0a24e0 legacy=0x7f1ffc1cabc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient: _finish_hunting 0 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 monclient: found mon.a 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ffc09d180 con 0x7f1ffc13c220 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient: _finish_auth 0 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient: _check_auth_tickets 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 5 monclient: authenticate success, global_id 114168 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: _renew_subs 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:20.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1ffc138280 con 0x7f1ffc13c220 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 client.114168.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: _renew_subs 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1ffc137580 con 0x7f1ffc13c220 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f2001a31640 1 librados: init done 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1ff4004fc0 con 0x7f1ffc13c220 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1ff4005310 con 0x7f1ffc13c220 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1fe3fff640 10 monclient: dump: 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:20.839 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.821+0000 7f1ff8ff9640 4 set_mon_vals no callback set 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f2001a31640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f2001a31640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f2001a31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1ff4011380 con 0x7f1ffc13c220 2024-09-15T20:41:20.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fdc005090 con 0x7f1ffc13c220 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7f1ff404ba80 con 0x7f1ffc13c220 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f1fe3fff640 10 client.114168.objecter ms_dispatch 0x7f1ffc000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f1fe3fff640 3 client.114168.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f1fe3fff640 3 client.114168.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f1fe3fff640 20 client.114168.objecter dump_active .. 0 homeless 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f1ffaffd640 10 monclient: get_auth_request con 0x7f1fe803ded0 auth_method 0 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.825+0000 7f1fe3fff640 10 client.114168.objecter ms_handle_connect 0x7f1fe803ded0 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.829+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1ff4016f00 con 0x7f1ffc13c220 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.829+0000 7f1fe3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:20.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:20.829+0000 7f1fe3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:21.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:41:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1fdc005380 con 0x7f1ffc13c220 2024-09-15T20:41:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f1fe3fff640 1 -- 172.21.15.78:0/4245127788 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3264728901) 0x7f1ff4016800 con 0x7f1ffc13c220 2024-09-15T20:41:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f1fe3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:21.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f1fe3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:21.070 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":38,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:41:21.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 10 librados: watch_flush enter 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 10 librados: watch_flush exit 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 20 client.114168.objecter shutdown clearing up homeless session... 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 10 client.114168.objecter successfully canceled tick 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 >> v1:172.21.15.146:6812/435598642 conn(0x7f1fe803ded0 legacy=0x7f1fe8040390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 10 monclient: shutdown 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 >> v1:172.21.15.78:6789/0 conn(0x7f1ffc13c220 legacy=0x7f1ffc1d2e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 shutdown_connections 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.065+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 >> 172.21.15.78:0/4245127788 conn(0x7f1ffc0a63c0 msgr2=0x7f1ffc12e900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.069+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 shutdown_connections 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.069+0000 7f2001a31640 1 -- 172.21.15.78:0/4245127788 wait complete. 2024-09-15T20:41:21.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:21.069+0000 7f2001a31640 1 librados: shutdown 2024-09-15T20:41:24.088 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:41:24.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient: get_monmap_and_config 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient: build_initial_monmap 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient: monmap: 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:24.218149+0000 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:24.218149+0000 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:24.217 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 1 Processor -- start 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 1 -- start start 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient: init 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient(hunting): picked mon.noname-a con 0x7f6218135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient(hunting): picked mon.noname-c con 0x7f62180a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient(hunting): picked mon.noname-b con 0x7f621813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f62180745e0 con 0x7f6218135b60 2024-09-15T20:41:24.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621806ecf0 con 0x7f62180a24e0 2024-09-15T20:41:24.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621806efd0 con 0x7f621813c220 2024-09-15T20:41:24.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:24.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621d029640 10 monclient(hunting): authenticate will time out at 1745.102925s 2024-09-15T20:41:24.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f621759e640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f621813e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:24.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f6216d9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6218135b60 0x7f6218135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:36656/0 (socket says 172.21.15.78:36656) 2024-09-15T20:41:24.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.213+0000 7f6216d9d640 1 -- 172.21.15.78:0/2934452723 learned_addr learned my addr 172.21.15.78:0/2934452723 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:24.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.417+0000 7f621759e640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f621813e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:24.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:24.821+0000 7f621759e640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f621813e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:25.624 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:25.625+0000 7f621759e640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f621813e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 10 monclient(hunting): tick 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 1 monclient(hunting): continuing hunt 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f621813e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.146:6789/0 conn(0x7f62180a24e0 legacy=0x7f621813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6789/0 conn(0x7f6218135b60 legacy=0x7f6218135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 10 monclient(hunting): picked mon.noname-b con 0x7f621813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 10 monclient(hunting): picked mon.noname-a con 0x7f61f8006e60 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:27.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 10 monclient(hunting): picked mon.noname-c con 0x7f62180a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:27.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6218136650 con 0x7f61f8006e60 2024-09-15T20:41:27.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621806efd0 con 0x7f62180a24e0 2024-09-15T20:41:27.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f80103d0 con 0x7f621813c220 2024-09-15T20:41:27.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6214d99640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:27.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.213+0000 7f6216d9d640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:27.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.417+0000 7f6216d9d640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:27.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:27.821+0000 7f6216d9d640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:28.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:28.624+0000 7f6216d9d640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:30.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 10 monclient(hunting): tick 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 1 monclient(hunting): continuing hunt 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.146:6789/0 conn(0x7f62180a24e0 legacy=0x7f61f800cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6789/0 conn(0x7f61f8006e60 legacy=0x7f61f80092d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 10 monclient(hunting): picked mon.noname-a con 0x7f61f80106d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 10 monclient(hunting): picked mon.noname-b con 0x7f621813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 10 monclient(hunting): picked mon.noname-c con 0x7f62180a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f80116d0 con 0x7f61f80106d0 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f80103d0 con 0x7f62180a24e0 2024-09-15T20:41:30.220 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 1 -- 172.21.15.78:0/2934452723 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f61f80082d0 con 0x7f621813c220 2024-09-15T20:41:30.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f6214d99640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:30.221 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.212+0000 7f621659c640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:30.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.416+0000 7f621659c640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:30.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:30.820+0000 7f621659c640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2144097973 0 0) 0x7f61f80103d0 con 0x7f62180a24e0 2024-09-15T20:41:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient(hunting): my global_id is 23653 2024-09-15T20:41:31.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6204003530 con 0x7f62180a24e0 2024-09-15T20:41:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1226652794 0 0) 0x7f6204003530 con 0x7f62180a24e0 2024-09-15T20:41:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6790/0 conn(0x7f621813c220 legacy=0x7f61f8005720 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:41:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.78:6789/0 conn(0x7f61f80106d0 legacy=0x7f61f8010ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: _finish_hunting 0 2024-09-15T20:41:31.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 monclient: found mon.noname-c 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61f80094b0 con 0x7f62180a24e0 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: _finish_auth 0 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: _check_auth_tickets 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6210004520 con 0x7f62180a24e0 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: dump: 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.400 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f621d029640 5 monclient: authenticate success, global_id 23653 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f621d029640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f62100046c0 con 0x7f62180a24e0 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f621d029640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:31.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f621d029640 4 set_mon_vals no callback set 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 1 -- 172.21.15.78:0/2934452723 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6210004b50 con 0x7f62180a24e0 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.392+0000 7f6215d9b640 10 monclient: dump: 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.402 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient: shutdown 2024-09-15T20:41:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- 172.21.15.78:0/2934452723 >> v1:172.21.15.146:6789/0 conn(0x7f62180a24e0 legacy=0x7f61f800cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- 172.21.15.78:0/2934452723 shutdown_connections 2024-09-15T20:41:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- 172.21.15.78:0/2934452723 >> 172.21.15.78:0/2934452723 conn(0x7f62180a63c0 msgr2=0x7f62180a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- 172.21.15.78:0/2934452723 shutdown_connections 2024-09-15T20:41:31.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- 172.21.15.78:0/2934452723 wait complete. 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient: build_initial_monmap 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient: monmap: 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 librados: starting msgr at 2024-09-15T20:41:31.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 librados: starting objecter 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 Processor -- start 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- start start 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 librados: setting wanted keys 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 librados: calling monclient init 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient: init 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient(hunting): picked mon.noname-a con 0x7f61f80106d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient(hunting): picked mon.noname-c con 0x7f62180a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient(hunting): picked mon.noname-b con 0x7f621813c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6218121400 con 0x7f61f80106d0 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621809d180 con 0x7f621813c220 2024-09-15T20:41:31.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f621811f0a0 con 0x7f62180a24e0 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621d029640 10 monclient(hunting): authenticate will time out at 1752.287256s 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621659c640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f62180a24e0 legacy=0x7f62181d1aa0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621759e640 1 --1- >> v1:172.21.15.146:6789/0 conn(0x7f621813c220 0x7f62181d51d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.146:6789/0 says I am v1:172.21.15.78:45272/0 (socket says 172.21.15.78:45272) 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f621759e640 1 -- 172.21.15.78:0/3591568422 learned_addr learned my addr 172.21.15.78:0/3591568422 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f620f7fe640 10 client.?.objecter ms_handle_connect 0x7f621813c220 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.396+0000 7f620f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2689799209 0 0) 0x7f621809d180 con 0x7f621813c220 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient(hunting): my global_id is 23656 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f61f80103d0 con 0x7f621813c220 2024-09-15T20:41:31.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 717160780 0 0) 0x7f61f80103d0 con 0x7f621813c220 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f621809d180 con 0x7f621813c220 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6210004a10 con 0x7f621813c220 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient(hunting): dump: 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.410 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1302567764 0 0) 0x7f621809d180 con 0x7f621813c220 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 >> v1:172.21.15.78:6790/0 conn(0x7f62180a24e0 legacy=0x7f62181d1aa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 >> v1:172.21.15.78:6789/0 conn(0x7f61f80106d0 legacy=0x7f6218133a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 monclient: found mon.b 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6218121120 con 0x7f621813c220 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:31.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 5 monclient: authenticate success, global_id 23656 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 10 monclient: _renew_subs 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f621811fc00 con 0x7f621813c220 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 10 client.23656.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 10 monclient: _renew_subs 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f6218134300 con 0x7f621813c220 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 1 librados: init done 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6210004e60 con 0x7f621813c220 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6210005110 con 0x7f621813c220 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f6214d99640 4 set_mon_vals no callback set 2024-09-15T20:41:31.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 monclient: dump: 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:31.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:41:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61fc005090 con 0x7f621813c220 2024-09-15T20:41:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f6210005330 con 0x7f621813c220 2024-09-15T20:41:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7f621004bac0 con 0x7f621813c220 2024-09-15T20:41:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 client.23656.objecter ms_dispatch 0x7f6218000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 3 client.23656.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 3 client.23656.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:31.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 20 client.23656.objecter dump_active .. 0 homeless 2024-09-15T20:41:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f6216d9d640 10 monclient: get_auth_request con 0x7f61f803ef40 auth_method 0 2024-09-15T20:41:31.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.400+0000 7f620f7fe640 10 client.23656.objecter ms_handle_connect 0x7f61f803ef40 2024-09-15T20:41:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.420+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f6210016fa0 con 0x7f621813c220 2024-09-15T20:41:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.420+0000 7f620f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:31.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.420+0000 7f620f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.656+0000 7f621d029640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:41:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.656+0000 7f621d029640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.656+0000 7f621d029640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:41:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.656+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f61fc005380 con 0x7f621813c220 2024-09-15T20:41:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.656+0000 7f620f7fe640 1 -- 172.21.15.78:0/3591568422 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 464256763) 0x7f62100168a0 con 0x7f621813c220 2024-09-15T20:41:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.656+0000 7f620f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:41:31.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.656+0000 7f620f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:31.663 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":42,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:41:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 10 librados: watch_flush enter 2024-09-15T20:41:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 10 librados: watch_flush exit 2024-09-15T20:41:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 20 client.23656.objecter shutdown clearing up homeless session... 2024-09-15T20:41:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 10 client.23656.objecter successfully canceled tick 2024-09-15T20:41:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 >> v1:172.21.15.146:6812/435598642 conn(0x7f61f803ef40 legacy=0x7f61f8041400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 10 monclient: shutdown 2024-09-15T20:41:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:31.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 >> v1:172.21.15.146:6789/0 conn(0x7f621813c220 legacy=0x7f62181d51d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 shutdown_connections 2024-09-15T20:41:31.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 >> 172.21.15.78:0/3591568422 conn(0x7f62180a63c0 msgr2=0x7f6218134cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:31.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 shutdown_connections 2024-09-15T20:41:31.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 1 -- 172.21.15.78:0/3591568422 wait complete. 2024-09-15T20:41:31.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.660+0000 7f621d029640 1 librados: shutdown 2024-09-15T20:41:31.683 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:41:31.683 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T20:41:31.683 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:41:31.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: get_monmap_and_config 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: build_initial_monmap 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: monmap: 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:31.801268+0000 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:31.801268+0000 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:31.800 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:31.801 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 Processor -- start 2024-09-15T20:41:31.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- start start 2024-09-15T20:41:31.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: init 2024-09-15T20:41:31.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:31.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient(hunting): picked mon.noname-a con 0x7f78d8137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient(hunting): picked mon.noname-b con 0x7f78d8074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient(hunting): picked mon.noname-c con 0x7f78d813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d8138850 con 0x7f78d8137d60 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d80a5da0 con 0x7f78d813e370 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d8073560 con 0x7f78d8074500 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient(hunting): authenticate will time out at 1752.685887s 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d7fff640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f78d8074500 legacy=0x7f78d813cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78dc932640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f78d8137d60 0x7f78d8138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58862/0 (socket says 172.21.15.78:58862) 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78dc932640 1 -- 172.21.15.78:0/1567084708 learned_addr learned my addr 172.21.15.78:0/1567084708 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3976776876 0 0) 0x7f78d8138850 con 0x7f78d8137d60 2024-09-15T20:41:31.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient(hunting): my global_id is 114171 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c0003530 con 0x7f78d8137d60 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 89024447 0 0) 0x7f78c0003530 con 0x7f78d8137d60 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 >> v1:172.21.15.78:6790/0 conn(0x7f78d8074500 legacy=0x7f78d813cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 >> v1:172.21.15.146:6789/0 conn(0x7f78d813e370 legacy=0x7f78d8140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 monclient: found mon.noname-a 2024-09-15T20:41:31.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d8073840 con 0x7f78d8137d60 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: _finish_auth 0 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f78cc0016a0 con 0x7f78d8137d60 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: dump: 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.806 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 5 monclient: authenticate success, global_id 114171 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78cc0018c0 con 0x7f78d8137d60 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 1 -- 172.21.15.78:0/1567084708 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f78cc001cd0 con 0x7f78d8137d60 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78d77fe640 10 monclient: dump: 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.807 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 4 set_mon_vals no callback set 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: shutdown 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:31.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- 172.21.15.78:0/1567084708 >> v1:172.21.15.78:6789/0 conn(0x7f78d8137d60 legacy=0x7f78d8138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- 172.21.15.78:0/1567084708 shutdown_connections 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- 172.21.15.78:0/1567084708 >> 172.21.15.78:0/1567084708 conn(0x7f78d80aa650 msgr2=0x7f78d80aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- 172.21.15.78:0/1567084708 shutdown_connections 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 1 -- 172.21.15.78:0/1567084708 wait complete. 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: build_initial_monmap 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.796+0000 7f78deb96640 10 monclient: monmap: 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:31.809 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:31.810 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 librados: starting msgr at 2024-09-15T20:41:31.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 librados: starting objecter 2024-09-15T20:41:31.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 Processor -- start 2024-09-15T20:41:31.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 -- start start 2024-09-15T20:41:31.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 librados: setting wanted keys 2024-09-15T20:41:31.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 librados: calling monclient init 2024-09-15T20:41:31.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient: init 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient(hunting): picked mon.noname-a con 0x7f78d8074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient(hunting): picked mon.noname-c con 0x7f78d8137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient(hunting): picked mon.noname-b con 0x7f78d813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d813f990 con 0x7f78d8074500 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d8140400 con 0x7f78d813e370 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78d8141420 con 0x7f78d8137d60 2024-09-15T20:41:31.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient(hunting): authenticate will time out at 1752.688759s 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d7fff640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f78d8137d60 legacy=0x7f78d813f280 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78dc932640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f78d8074500 0x7f78d813eb70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58874/0 (socket says 172.21.15.78:58874) 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78dc932640 1 -- 172.21.15.78:0/2923089748 learned_addr learned my addr 172.21.15.78:0/2923089748 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 client.?.objecter ms_handle_connect 0x7f78d8074500 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 158165877 0 0) 0x7f78d813f990 con 0x7f78d8074500 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): my global_id is 114174 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c4004ec0 con 0x7f78d8074500 2024-09-15T20:41:31.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 client.?.objecter ms_handle_connect 0x7f78d813e370 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1634980396 0 0) 0x7f78c4004ec0 con 0x7f78d8074500 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78d813f990 con 0x7f78d8074500 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f78cc001c10 con 0x7f78d8074500 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): dump: 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.817 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3258024176 0 0) 0x7f78d8140400 con 0x7f78d813e370 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): my global_id is 23662 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78c4004ec0 con 0x7f78d813e370 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 355084668 0 0) 0x7f78d813f990 con 0x7f78d8074500 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 >> v1:172.21.15.78:6790/0 conn(0x7f78d8137d60 legacy=0x7f78d813f280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 >> v1:172.21.15.146:6789/0 conn(0x7f78d813e370 legacy=0x7f78d81d4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: _finish_hunting 0 2024-09-15T20:41:31.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 monclient: found mon.a 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d8132420 con 0x7f78d8074500 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: _finish_auth 0 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: _check_auth_tickets 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 5 monclient: authenticate success, global_id 114174 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient: _renew_subs 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f78d8132760 con 0x7f78d8074500 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 client.114174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient: _renew_subs 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:31.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f78d8131e50 con 0x7f78d8074500 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 librados: init done 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f78cc004c00 con 0x7f78d8074500 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f78cc004db0 con 0x7f78d8074500 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d6ffd640 4 set_mon_vals no callback set 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: dump: 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.820 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78b80065d0 con 0x7f78b8002870 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f78cc004fd0 con 0x7f78d8074500 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 client.114174.objecter ms_handle_connect 0x7f78b8002870 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 client.114174.objecter resend_mon_ops 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7f78cc04b810 con 0x7f78d8074500 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 client.114174.objecter ms_dispatch 0x7f78d8000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:31.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 3 client.114174.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 3 client.114174.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 20 client.114174.objecter dump_active .. 0 homeless 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 548238475 0 0) 0x7f78b80065d0 con 0x7f78b8002870 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient(hunting): my global_id is 114177 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78d813f990 con 0x7f78b8002870 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78dd133640 10 monclient: get_auth_request con 0x7f78c403e030 auth_method 0 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2937903664 0 0) 0x7f78d813f990 con 0x7f78b8002870 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78b80065d0 con 0x7f78b8002870 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f78c8003ef0 con 0x7f78b8002870 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: dump: 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1014103564 0 0) 0x7f78b80065d0 con 0x7f78b8002870 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78b800a170 con 0x7f78b8002870 2024-09-15T20:41:31.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 client.114174.objecter ms_handle_connect 0x7f78c403e030 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f78c8006310 con 0x7f78b8002870 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 >> v1:172.21.15.78:6789/0 conn(0x7f78b8002870 legacy=0x7f78b8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.800+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 >> v1:172.21.15.78:6789/0 conn(0x7f78b8002870 legacy=0x7f78b8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f78b8009e60 con 0x7f78b8009120 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 client.114174.objecter ms_handle_connect 0x7f78b8009120 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 client.114174.objecter resend_mon_ops 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1170027688 0 0) 0x7f78b8009e60 con 0x7f78b8009120 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient(hunting): my global_id is 114180 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78b80065d0 con 0x7f78b8009120 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2922229626 0 0) 0x7f78b80065d0 con 0x7f78b8009120 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f78b8009e60 con 0x7f78b8009120 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f78c80045e0 con 0x7f78b8009120 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient: dump: 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.825 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2605444894 0 0) 0x7f78b8009e60 con 0x7f78b8009120 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f78b800c400 con 0x7f78b8009120 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4128 (unknown 0 0 3477729742) 0x7f78c8005a90 con 0x7f78b8009120 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 >> v1:172.21.15.78:6789/0 conn(0x7f78b8009120 legacy=0x7f78b80095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78d57fa640 1 -- 172.21.15.78:0/2923089748 >> v1:172.21.15.78:6789/0 conn(0x7f78b8009120 legacy=0x7f78b80095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 42, 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17651, 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:41:31.827 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:31.828 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:31.829 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:41:31.830 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:31.831 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.832 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:31.833 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:31.834 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:41:31.835 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:31.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.837 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.838 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:41:31.838 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:41:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 10 librados: watch_flush enter 2024-09-15T20:41:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 10 librados: watch_flush exit 2024-09-15T20:41:31.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 20 client.114174.objecter shutdown clearing up homeless session... 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 10 client.114174.objecter successfully canceled tick 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 >> v1:172.21.15.146:6812/435598642 conn(0x7f78c403e030 legacy=0x7f78c40404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 10 monclient: shutdown 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 >> v1:172.21.15.78:6789/0 conn(0x7f78d8074500 legacy=0x7f78d813eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78dd133640 1 -- 172.21.15.78:0/2923089748 reap_dead start 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78dd133640 1 -- 172.21.15.78:0/2923089748 reap_dead start 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 shutdown_connections 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 >> 172.21.15.78:0/2923089748 conn(0x7f78d80aa650 msgr2=0x7f78d80a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 shutdown_connections 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 1 -- 172.21.15.78:0/2923089748 wait complete. 2024-09-15T20:41:31.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.808+0000 7f78deb96640 1 librados: shutdown 2024-09-15T20:41:31.841 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:41:31.841 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T20:41:31.841 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:41:31.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient: get_monmap_and_config 2024-09-15T20:41:31.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient: build_initial_monmap 2024-09-15T20:41:31.945 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient: monmap: 2024-09-15T20:41:31.945 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:31.945 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.945 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:41:31.946770+0000 2024-09-15T20:41:31.945 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:41:31.946770+0000 2024-09-15T20:41:31.946 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:31.946 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.946 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:31.946 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:41:31.946 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:41:31.946 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 1 Processor -- start 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 1 -- start start 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient: init 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient(hunting): picked mon.noname-b con 0x7f15a80a3ea0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient(hunting): picked mon.noname-a con 0x7f15a809cdf0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient(hunting): picked mon.noname-c con 0x7f15a80742d0 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:31.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15a659c640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f15a80a3ea0 legacy=0x7f15a8073bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15a8074780 con 0x7f15a809cdf0 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15a80a00d0 con 0x7f15a80742d0 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15a80a03b0 con 0x7f15a80a3ea0 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15acc9b640 10 monclient(hunting): authenticate will time out at 1752.831744s 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15a5d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f15a809cdf0 0x7f15a813a9b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58902/0 (socket says 172.21.15.78:58902) 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.940+0000 7f15a5d9b640 1 -- 172.21.15.78:0/2089336082 learned_addr learned my addr 172.21.15.78:0/2089336082 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2801213816 0 0) 0x7f15a8074780 con 0x7f15a809cdf0 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient(hunting): my global_id is 114183 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f158c003530 con 0x7f15a809cdf0 2024-09-15T20:41:31.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 232471018 0 0) 0x7f158c003530 con 0x7f15a809cdf0 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 >> v1:172.21.15.78:6790/0 conn(0x7f15a80a3ea0 legacy=0x7f15a8073bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 >> v1:172.21.15.146:6789/0 conn(0x7f15a80742d0 legacy=0x7f15a813e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: _finish_hunting 0 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 monclient: found mon.noname-a 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a8135b50 con 0x7f15a809cdf0 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: _finish_auth 0 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: _check_auth_tickets 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f15940016a0 con 0x7f15a809cdf0 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:31.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: dump: 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 5 monclient: authenticate success, global_id 114183 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1594001970 con 0x7f15a809cdf0 2024-09-15T20:41:31.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 1 -- 172.21.15.78:0/2089336082 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f15940048b0 con 0x7f15a809cdf0 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: get_monmap_and_config success 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 4 set_mon_vals no callback set 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a559a640 10 monclient: dump: 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.953 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: shutdown 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- 172.21.15.78:0/2089336082 >> v1:172.21.15.78:6789/0 conn(0x7f15a809cdf0 legacy=0x7f15a813a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- 172.21.15.78:0/2089336082 shutdown_connections 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- 172.21.15.78:0/2089336082 >> 172.21.15.78:0/2089336082 conn(0x7f15a80a3380 msgr2=0x7f15a80a3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- 172.21.15.78:0/2089336082 shutdown_connections 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- 172.21.15.78:0/2089336082 wait complete. 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: build_initial_monmap 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: monmap: 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:41:31.954 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:41:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:41:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:41:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:41:31.955 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:41:31.955 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 librados: starting msgr at 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 librados: starting objecter 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 Processor -- start 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- start start 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 librados: setting wanted keys 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 librados: calling monclient init 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: init 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: _reopen_session rank -1 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient(hunting): picked mon.noname-c con 0x7f15a80742d0 addr v1:172.21.15.78:6790/0 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient(hunting): picked mon.noname-a con 0x7f15a80a3ea0 addr v1:172.21.15.78:6789/0 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a659c640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f15a80742d0 legacy=0x7f15a8072410 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient(hunting): picked mon.noname-b con 0x7f15a8137440 addr v1:172.21.15.146:6789/0 2024-09-15T20:41:31.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15a8138c40 con 0x7f15a80a3ea0 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15a8073230 con 0x7f15a8137440 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15a8136a50 con 0x7f15a80742d0 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient(hunting): _renew_subs 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient(hunting): authenticate will time out at 1752.835178s 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a5d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f15a80a3ea0 0x7f15a8072b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58904/0 (socket says 172.21.15.78:58904) 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15a5d9b640 1 -- 172.21.15.78:0/314611513 learned_addr learned my addr 172.21.15.78:0/314611513 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 client.?.objecter ms_handle_connect 0x7f15a80a3ea0 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 217447046 0 0) 0x7f15a8138c40 con 0x7f15a80a3ea0 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient(hunting): my global_id is 114186 2024-09-15T20:41:31.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1590004ec0 con 0x7f15a80a3ea0 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 client.?.objecter ms_handle_connect 0x7f15a8137440 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2440602030 0 0) 0x7f1590004ec0 con 0x7f15a80a3ea0 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15a8138c40 con 0x7f15a80a3ea0 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f15940014c0 con 0x7f15a80a3ea0 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient(hunting): dump: 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.962 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.963 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.963 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.963 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.963 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.963 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.963 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:41:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2010159933 0 0) 0x7f15a8138c40 con 0x7f15a80a3ea0 2024-09-15T20:41:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 >> v1:172.21.15.78:6790/0 conn(0x7f15a80742d0 legacy=0x7f15a8072410 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:41:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 >> v1:172.21.15.146:6789/0 conn(0x7f15a8137440 legacy=0x7f15a81d2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient: _finish_hunting 0 2024-09-15T20:41:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 monclient: found mon.a 2024-09-15T20:41:31.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15a8074d10 con 0x7f15a80a3ea0 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient: _finish_auth 0 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 10 monclient: _check_auth_tickets 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f159effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 5 monclient: authenticate success, global_id 114186 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: _renew_subs 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:31.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f15a8137dd0 con 0x7f15a80a3ea0 2024-09-15T20:41:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 client.114186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:41:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: _renew_subs 2024-09-15T20:41:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:41:31.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f15a813e4e0 con 0x7f15a80a3ea0 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.944+0000 7f15acc9b640 1 librados: init done 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1594001830 con 0x7f15a80a3ea0 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1594004fc0 con 0x7f15a80a3ea0 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f15a4d99640 4 set_mon_vals no callback set 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: dump: 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.967 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f15acc9b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f15840065d0 con 0x7f1584002870 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1594011030 con 0x7f15a80a3ea0 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 client.114186.objecter ms_handle_connect 0x7f1584002870 2024-09-15T20:41:31.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 client.114186.objecter resend_mon_ops 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(380..380 src has 223..380) ==== 4961+0+0 (unknown 2582926993 0 0) 0x7f159404b730 con 0x7f15a80a3ea0 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 client.114186.objecter ms_dispatch 0x7f15a8000ba0 osd_map(380..380 src has 223..380) 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 3 client.114186.objecter handle_osd_map got epochs [380,380] > 0 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 3 client.114186.objecter handle_osd_map decoding full epoch 380 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 20 client.114186.objecter dump_active .. 0 homeless 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1815803035 0 0) 0x7f15840065d0 con 0x7f1584002870 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient(hunting): my global_id is 23668 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15a8138c40 con 0x7f1584002870 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f15a6d9d640 10 monclient: get_auth_request con 0x7f159003dee0 auth_method 0 2024-09-15T20:41:31.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3767384593 0 0) 0x7f15a8138c40 con 0x7f1584002870 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15840065d0 con 0x7f1584002870 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f15a00042f0 con 0x7f1584002870 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: dump: 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.970 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 client.114186.objecter ms_handle_connect 0x7f159003dee0 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2899588965 0 0) 0x7f15840065d0 con 0x7f1584002870 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f158400a170 con 0x7f1584002870 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f15a0006650 con 0x7f1584002870 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 >> v1:172.21.15.146:6789/0 conn(0x7f1584002870 legacy=0x7f1584004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.948+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 >> v1:172.21.15.146:6789/0 conn(0x7f1584002870 legacy=0x7f1584004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:41:31.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1584009e60 con 0x7f1584009120 2024-09-15T20:41:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 client.114186.objecter ms_handle_connect 0x7f1584009120 2024-09-15T20:41:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 client.114186.objecter resend_mon_ops 2024-09-15T20:41:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2314295892 0 0) 0x7f1584009e60 con 0x7f1584009120 2024-09-15T20:41:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient(hunting): _negotiate 2024-09-15T20:41:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:41:31.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient(hunting): my global_id is 23671 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f15840065d0 con 0x7f1584009120 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1677104112 0 0) 0x7f15840065d0 con 0x7f1584009120 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1584009e60 con 0x7f1584009120 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f15a0004960 con 0x7f1584009120 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient: dump: 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:41:31.974 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3864807808 0 0) 0x7f1584009e60 con 0x7f1584009120 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f158400c400 con 0x7f1584009120 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4294 (unknown 0 0 56972319) 0x7f15a0005ef0 con 0x7f1584009120 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 >> v1:172.21.15.146:6789/0 conn(0x7f1584009120 legacy=0x7f15840095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f159effd640 1 -- 172.21.15.78:0/314611513 >> v1:172.21.15.146:6789/0 conn(0x7f1584009120 legacy=0x7f15840095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:41:31.976 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:41:31.976 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:31.976 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:31.976 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:41:31.976 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 42, 2024-09-15T20:41:31.976 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 604465, 2024-09-15T20:41:31.976 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:41:31.977 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:41:31.977 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-09-15T20:41:31.977 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.977 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:41:31.977 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:31.977 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:41:31.977 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:41:31.978 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.979 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:41:31.980 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:41:31.981 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:31.982 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:41:31.983 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.984 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:31.985 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.986 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:31.987 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:31.988 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:41:31.989 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 10 librados: watch_flush enter 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 10 librados: watch_flush exit 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 20 client.114186.objecter shutdown clearing up homeless session... 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 10 client.114186.objecter successfully canceled tick 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 >> v1:172.21.15.146:6812/435598642 conn(0x7f159003dee0 legacy=0x7f15900403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 10 monclient: shutdown 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 >> v1:172.21.15.78:6789/0 conn(0x7f15a80a3ea0 legacy=0x7f15a8072b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:41:31.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15a6d9d640 1 -- 172.21.15.78:0/314611513 reap_dead start 2024-09-15T20:41:31.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15a6d9d640 1 -- 172.21.15.78:0/314611513 reap_dead start 2024-09-15T20:41:31.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 shutdown_connections 2024-09-15T20:41:31.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 >> 172.21.15.78:0/314611513 conn(0x7f15a80a3380 msgr2=0x7f15a80a47f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:41:31.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 shutdown_connections 2024-09-15T20:41:31.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 1 -- 172.21.15.78:0/314611513 wait complete. 2024-09-15T20:41:31.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:41:31.956+0000 7f15acc9b640 1 librados: shutdown 2024-09-15T20:41:31.993 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:41:31.993 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:42:01.249 INFO:tasks.workunit.client.0.smithi146.stdout: OK ] LibRadosTwoPoolsPP.ProxyRead (18040 ms) 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.CachePin 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.CachePin (23025 ms) 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.SetRedirectRead 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.SetRedirectRead (3032 ms) 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestPromoteRead 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestPromoteRead (3026 ms) 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRefRead 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRefRead (2898 ms) 2024-09-15T20:42:01.250 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestUnset 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestUnset (3025 ms) 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestDedupRefRead 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestDedupRefRead (4030 ms) 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount (45561 ms) 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapRefcount2 (31294 ms) 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestTestSnapCreate 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.251 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestTestSnapCreate (3117 ms) 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRedirectAfterPromote (3044 ms) 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestCheckRefcountWhenModification (24202 ms) 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapIncCount 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapIncCount (14788 ms) 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvict 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.252 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvict (4969 ms) 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictPromote 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictPromote (4035 ms) 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: waiting for scrubs... 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: done waiting 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapSizeMismatch (24640 ms) 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.DedupFlushRead 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.DedupFlushRead (24906 ms) 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushSnap 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.253 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushSnap (8981 ms) 2024-09-15T20:42:01.254 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestFlushDupCount 2024-09-15T20:42:01.254 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:42:01.254 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestFlushDupCount (9000 ms) 2024-09-15T20:42:01.254 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringFlush 2024-09-15T20:42:58.304 INFO:tasks.ceph.osd.3.smithi146.stderr:2024-09-15T20:42:58.302+0000 7fae752b3640 -1 snap_mapper.add_oid found existing snaps mapped on 102:28109b5c:test-rados-api-smithi146-28320-97::foo:21, removing 2024-09-15T20:42:58.305 INFO:tasks.ceph.osd.4.smithi146.stderr:2024-09-15T20:42:58.302+0000 7f195e5c8640 -1 snap_mapper.add_oid found existing snaps mapped on 102:28109b5c:test-rados-api-smithi146-28320-97::foo:21, removing 2024-09-15T20:43:01.994 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T20:43:01.995 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T20:43:01.995 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T20:43:01.998 INFO:tasks.ceph.mon.c:Started 2024-09-15T20:43:01.998 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:43:01.998 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:43:02.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient: get_monmap_and_config 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient: monmap: 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:02.119394+0000 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:02.119394+0000 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.118 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:02.119 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:02.119 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 1 Processor -- start 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 1 -- start start 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient: init 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient(hunting): picked mon.noname-a con 0x7feb50137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient(hunting): picked mon.noname-c con 0x7feb500a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient(hunting): picked mon.noname-b con 0x7feb5013e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb500a6420 con 0x7feb50137ce0 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb500a6700 con 0x7feb500a9610 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb500a2110 con 0x7feb5013e5f0 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb56bd4640 10 monclient(hunting): authenticate will time out at 1843.003886s 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.115+0000 7feb55171640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7feb5013e5f0 0x7feb501409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50798/0 (socket says 172.21.15.78:50798) 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb55171640 1 -- 172.21.15.78:0/1923718028 learned_addr learned my addr 172.21.15.78:0/1923718028 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4189747737 0 0) 0x7feb500a6420 con 0x7feb50137ce0 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient(hunting): my global_id is 114189 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb40003530 con 0x7feb50137ce0 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2296033857 0 0) 0x7feb40003530 con 0x7feb50137ce0 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 >> v1:172.21.15.78:6790/0 conn(0x7feb5013e5f0 legacy=0x7feb501409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 >> v1:172.21.15.146:6789/0 conn(0x7feb500a9610 legacy=0x7feb5013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 monclient: found mon.noname-a 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb500a4bc0 con 0x7feb50137ce0 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feb440015d0 con 0x7feb50137ce0 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: dump: 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.129 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 5 monclient: authenticate success, global_id 114189 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb440017f0 con 0x7feb50137ce0 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 1 -- 172.21.15.78:0/1923718028 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feb44001bc0 con 0x7feb50137ce0 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4f7fe640 10 monclient: dump: 2024-09-15T20:43:02.130 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 4 set_mon_vals no callback set 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient: shutdown 2024-09-15T20:43:02.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- 172.21.15.78:0/1923718028 >> v1:172.21.15.78:6789/0 conn(0x7feb50137ce0 legacy=0x7feb501380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- 172.21.15.78:0/1923718028 shutdown_connections 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- 172.21.15.78:0/1923718028 >> 172.21.15.78:0/1923718028 conn(0x7feb500aa6c0 msgr2=0x7feb500aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- 172.21.15.78:0/1923718028 shutdown_connections 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- 172.21.15.78:0/1923718028 wait complete. 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient: monmap: 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:02.132 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 librados: starting msgr at 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 librados: starting objecter 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 Processor -- start 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- start start 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 librados: setting wanted keys 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 librados: calling monclient init 2024-09-15T20:43:02.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient: init 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient(hunting): picked mon.noname-c con 0x7feb500a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient(hunting): picked mon.noname-a con 0x7feb50137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient(hunting): picked mon.noname-b con 0x7feb5013e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb500a14b0 con 0x7feb50137ce0 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb5009e470 con 0x7feb5013e5f0 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feb500a07d0 con 0x7feb500a9610 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb56bd4640 10 monclient(hunting): authenticate will time out at 1843.007089s 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7feb50137ce0 0x7feb5009d3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55432/0 (socket says 172.21.15.78:55432) 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4ffff640 1 -- 172.21.15.78:0/150531510 learned_addr learned my addr 172.21.15.78:0/150531510 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.134 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb54970640 1 --1- 172.21.15.78:0/150531510 >> v1:172.21.15.78:6790/0 conn(0x7feb500a9610 0x7feb5009ccb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50808/0 (socket says 172.21.15.78:50808) 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 client.?.objecter ms_handle_connect 0x7feb500a9610 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 client.?.objecter ms_handle_connect 0x7feb50137ce0 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2450176993 0 0) 0x7feb500a14b0 con 0x7feb50137ce0 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): my global_id is 114192 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb34004ec0 con 0x7feb50137ce0 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 client.?.objecter ms_handle_connect 0x7feb5013e5f0 2024-09-15T20:43:02.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1090700320 0 0) 0x7feb34004ec0 con 0x7feb50137ce0 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feb500a14b0 con 0x7feb50137ce0 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feb3c0019a0 con 0x7feb50137ce0 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): dump: 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.136 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4085482761 0 0) 0x7feb5009e470 con 0x7feb5013e5f0 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 10 monclient(hunting): my global_id is 23677 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.119+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feb34004ec0 con 0x7feb5013e5f0 2024-09-15T20:43:02.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 582875895 0 0) 0x7feb500a14b0 con 0x7feb50137ce0 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 >> v1:172.21.15.78:6790/0 conn(0x7feb500a9610 legacy=0x7feb5009ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 >> v1:172.21.15.146:6789/0 conn(0x7feb5013e5f0 legacy=0x7feb501d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 monclient: found mon.a 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb501372e0 con 0x7feb50137ce0 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 5 monclient: authenticate success, global_id 114192 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 10 monclient: _renew_subs 2024-09-15T20:43:02.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feb5009efd0 con 0x7feb50137ce0 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 10 client.114192.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 10 monclient: _renew_subs 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7feb5009dad0 con 0x7feb50137ce0 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 1 librados: init done 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feb3c001ce0 con 0x7feb50137ce0 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feb3c003a70 con 0x7feb50137ce0 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4effd640 4 set_mon_vals no callback set 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 monclient: dump: 2024-09-15T20:43:02.139 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7feb3c003c10 con 0x7feb50137ce0 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.140 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb30005090 con 0x7feb50137ce0 2024-09-15T20:43:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(444..444 src has 223..444) ==== 5346+0+0 (unknown 2381257293 0 0) 0x7feb3c04b840 con 0x7feb50137ce0 2024-09-15T20:43:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 client.114192.objecter ms_dispatch 0x7feb50000ba0 osd_map(444..444 src has 223..444) 2024-09-15T20:43:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 3 client.114192.objecter handle_osd_map got epochs [444,444] > 0 2024-09-15T20:43:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 3 client.114192.objecter handle_osd_map decoding full epoch 444 2024-09-15T20:43:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 20 client.114192.objecter dump_active .. 0 homeless 2024-09-15T20:43:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb54970640 10 monclient: get_auth_request con 0x7feb3403e130 auth_method 0 2024-09-15T20:43:02.141 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.123+0000 7feb4d7fa640 10 client.114192.objecter ms_handle_connect 0x7feb3403e130 2024-09-15T20:43:02.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.127+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7feb3c016a80 con 0x7feb50137ce0 2024-09-15T20:43:02.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.127+0000 7feb4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.142 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.127+0000 7feb4d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:02.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.367+0000 7feb56bd4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:43:02.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.367+0000 7feb56bd4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:02.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.367+0000 7feb56bd4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.367+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7feb30005380 con 0x7feb50137ce0 2024-09-15T20:43:02.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.367+0000 7feb4d7fa640 1 -- 172.21.15.78:0/150531510 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3791129910) 0x7feb3c01ae60 con 0x7feb50137ce0 2024-09-15T20:43:02.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.367+0000 7feb4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:02.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.367+0000 7feb4d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:02.370 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:43:02.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 10 librados: watch_flush enter 2024-09-15T20:43:02.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 10 librados: watch_flush exit 2024-09-15T20:43:02.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 20 client.114192.objecter shutdown clearing up homeless session... 2024-09-15T20:43:02.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 10 client.114192.objecter successfully canceled tick 2024-09-15T20:43:02.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 >> v1:172.21.15.146:6812/435598642 conn(0x7feb3403e130 legacy=0x7feb340405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 10 monclient: shutdown 2024-09-15T20:43:02.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 >> v1:172.21.15.78:6789/0 conn(0x7feb50137ce0 legacy=0x7feb5009d3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 shutdown_connections 2024-09-15T20:43:02.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 >> 172.21.15.78:0/150531510 conn(0x7feb500aa6c0 msgr2=0x7feb500a2ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 shutdown_connections 2024-09-15T20:43:02.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 1 -- 172.21.15.78:0/150531510 wait complete. 2024-09-15T20:43:02.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.371+0000 7feb56bd4640 1 librados: shutdown 2024-09-15T20:43:02.387 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:43:02.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:43:02.387 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.491+0000 7f120b73c640 10 monclient: get_monmap_and_config 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.491+0000 7f120b73c640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.491+0000 7f120b73c640 10 monclient: monmap: 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:02.496581+0000 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:02.496581+0000 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.495 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.496 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:02.496 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:02.496 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 Processor -- start 2024-09-15T20:43:02.498 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- start start 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient: init 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient(hunting): picked mon.noname-a con 0x7f1204137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient(hunting): picked mon.noname-c con 0x7f1204074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient(hunting): picked mon.noname-b con 0x7f120413e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1204138850 con 0x7f1204137d60 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f12040a5da0 con 0x7f1204074500 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1204073560 con 0x7f120413e370 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient(hunting): authenticate will time out at 1843.381081s 2024-09-15T20:43:02.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1209cd9640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f120413e370 0x7f1204140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50816/0 (socket says 172.21.15.78:50816) 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1209cd9640 1 -- 172.21.15.78:0/3907271282 learned_addr learned my addr 172.21.15.78:0/3907271282 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1485735261 0 0) 0x7f1204138850 con 0x7f1204137d60 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient(hunting): my global_id is 114195 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11ec003530 con 0x7f1204137d60 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 449304665 0 0) 0x7f12040a5da0 con 0x7f1204074500 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient(hunting): my global_id is 23680 2024-09-15T20:43:02.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1204138850 con 0x7f1204074500 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3047152536 0 0) 0x7f1204138850 con 0x7f1204074500 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 >> v1:172.21.15.78:6790/0 conn(0x7f120413e370 legacy=0x7f1204140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 >> v1:172.21.15.78:6789/0 conn(0x7f1204137d60 legacy=0x7f1204138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 monclient: found mon.noname-c 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1204073840 con 0x7f1204074500 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f11f8001680 con 0x7f1204074500 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: dump: 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.502 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 5 monclient: authenticate success, global_id 23680 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11f80018a0 con 0x7f1204074500 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 1 -- 172.21.15.78:0/3907271282 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f11f8001cb0 con 0x7f1204074500 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 4 set_mon_vals no callback set 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f1203fff640 10 monclient: dump: 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.504 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 10 monclient: shutdown 2024-09-15T20:43:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- 172.21.15.78:0/3907271282 >> v1:172.21.15.146:6789/0 conn(0x7f1204074500 legacy=0x7f120413cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- 172.21.15.78:0/3907271282 shutdown_connections 2024-09-15T20:43:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- 172.21.15.78:0/3907271282 >> 172.21.15.78:0/3907271282 conn(0x7f12040aa650 msgr2=0x7f12040aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- 172.21.15.78:0/3907271282 shutdown_connections 2024-09-15T20:43:02.505 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.495+0000 7f120b73c640 1 -- 172.21.15.78:0/3907271282 wait complete. 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: monmap: 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.511 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 librados: starting msgr at 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 librados: starting objecter 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 Processor -- start 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 -- start start 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 librados: setting wanted keys 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 librados: calling monclient init 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: init 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:43:02.512 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient(hunting): picked mon.noname-a con 0x7f1204074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient(hunting): picked mon.noname-c con 0x7f1204137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient(hunting): picked mon.noname-b con 0x7f120413e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1204132bf0 con 0x7f1204074500 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1204136bb0 con 0x7f120413e370 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1204132ed0 con 0x7f1204137d60 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient(hunting): authenticate will time out at 1843.385186s 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1208cd7640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1204137d60 0x7f120409edf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50820/0 (socket says 172.21.15.78:50820) 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f12094d8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1204074500 0x7f120409e6e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55450/0 (socket says 172.21.15.78:55450) 2024-09-15T20:43:02.513 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1208cd7640 1 -- 172.21.15.78:0/1443526850 learned_addr learned my addr 172.21.15.78:0/1443526850 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.?.objecter ms_handle_connect 0x7f1204137d60 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.?.objecter ms_handle_connect 0x7f1204074500 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1200450629 0 0) 0x7f1204132bf0 con 0x7f1204074500 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): my global_id is 114198 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11f0004ec0 con 0x7f1204074500 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.?.objecter ms_handle_connect 0x7f120413e370 2024-09-15T20:43:02.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3563861446 0 0) 0x7f1204132ed0 con 0x7f1204137d60 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): my global_id is 124100 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1204132bf0 con 0x7f1204137d60 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 277779199 0 0) 0x7f11f0004ec0 con 0x7f1204074500 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1204132ed0 con 0x7f1204074500 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f11fc001960 con 0x7f1204074500 2024-09-15T20:43:02.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): dump: 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.516 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2117241043 0 0) 0x7f1204132bf0 con 0x7f1204137d60 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11f0004ec0 con 0x7f1204137d60 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f11f8001cb0 con 0x7f1204137d60 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): dump: 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.517 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 113761407 0 0) 0x7f1204132ed0 con 0x7f1204074500 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 >> v1:172.21.15.78:6790/0 conn(0x7f1204137d60 legacy=0x7f120409edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 >> v1:172.21.15.146:6789/0 conn(0x7f120413e370 legacy=0x7f120409f500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 monclient: found mon.a 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f120409e0b0 con 0x7f1204074500 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 5 monclient: authenticate success, global_id 114198 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: _renew_subs 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f12041418a0 con 0x7f1204074500 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 client.114198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: _renew_subs 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f12040a06c0 con 0x7f1204074500 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 librados: init done 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f11fc003620 con 0x7f1204074500 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f11fc003a70 con 0x7f1204074500 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f12037fe640 4 set_mon_vals no callback set 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: dump: 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.520 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f11e40065d0 con 0x7f11e4002870 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f11fc003c90 con 0x7f1204074500 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.114198.objecter ms_handle_connect 0x7f11e4002870 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.114198.objecter resend_mon_ops 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(445..445 src has 223..445) ==== 5354+0+0 (unknown 3722762973 0 0) 0x7f11fc04b770 con 0x7f1204074500 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 client.114198.objecter ms_dispatch 0x7f1204000ba0 osd_map(445..445 src has 223..445) 2024-09-15T20:43:02.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 3 client.114198.objecter handle_osd_map got epochs [445,445] > 0 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 3 client.114198.objecter handle_osd_map decoding full epoch 445 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 20 client.114198.objecter dump_active .. 0 homeless 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3135583326 0 0) 0x7f11e40065d0 con 0x7f11e4002870 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient(hunting): my global_id is 114201 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1204132ed0 con 0x7f11e4002870 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1082572231 0 0) 0x7f1204132ed0 con 0x7f11e4002870 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11e40065d0 con 0x7f11e4002870 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f11f80038c0 con 0x7f11e4002870 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 10 monclient: dump: 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1209cd9640 10 monclient: get_auth_request con 0x7f11f003e090 auth_method 0 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2775297005 0 0) 0x7f11e40065d0 con 0x7f11e4002870 2024-09-15T20:43:02.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.499+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11e400a170 con 0x7f11e4002870 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.503+0000 7f1201ffb640 10 client.114198.objecter ms_handle_connect 0x7f11f003e090 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.503+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f11f8005790 con 0x7f11e4002870 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.503+0000 7f1201ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.503+0000 7f1201ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.503+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 >> v1:172.21.15.78:6789/0 conn(0x7f11e4002870 legacy=0x7f11e4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.503+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 >> v1:172.21.15.78:6789/0 conn(0x7f11e4002870 legacy=0x7f11e4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f11e4009e60 con 0x7f11e4009120 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 client.114198.objecter ms_handle_connect 0x7f11e4009120 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 client.114198.objecter resend_mon_ops 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3225539578 0 0) 0x7f11e4009e60 con 0x7f11e4009120 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient(hunting): my global_id is 114204 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f11e40065d0 con 0x7f11e4009120 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2133224553 0 0) 0x7f11e40065d0 con 0x7f11e4009120 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f11e4009e60 con 0x7f11e4009120 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f11f8003e30 con 0x7f11e4009120 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient: dump: 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.525 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2029241400 0 0) 0x7f11e4009e60 con 0x7f11e4009120 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f11e400c400 con 0x7f11e4009120 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4140 (unknown 0 0 1020878746) 0x7f11f8005a00 con 0x7f11e4009120 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 >> v1:172.21.15.78:6789/0 conn(0x7f11e4009120 legacy=0x7f11e40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1201ffb640 1 -- 172.21.15.78:0/1443526850 >> v1:172.21.15.78:6789/0 conn(0x7f11e4009120 legacy=0x7f11e40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 44, 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 108350, 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:43:02.527 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.528 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.529 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.530 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:02.531 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.532 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.533 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:02.534 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.535 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.536 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.537 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.538 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:43:02.539 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.539 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.539 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.539 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:43:02.539 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 10 librados: watch_flush enter 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 10 librados: watch_flush exit 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 20 client.114198.objecter shutdown clearing up homeless session... 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 10 client.114198.objecter successfully canceled tick 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 >> v1:172.21.15.146:6812/435598642 conn(0x7f11f003e090 legacy=0x7f11f0040550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 10 monclient: shutdown 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 >> v1:172.21.15.78:6789/0 conn(0x7f1204074500 legacy=0x7f120409e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1209cd9640 1 -- 172.21.15.78:0/1443526850 reap_dead start 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f1209cd9640 1 -- 172.21.15.78:0/1443526850 reap_dead start 2024-09-15T20:43:02.541 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 shutdown_connections 2024-09-15T20:43:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 >> 172.21.15.78:0/1443526850 conn(0x7f12040aa650 msgr2=0x7f12040a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 shutdown_connections 2024-09-15T20:43:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 1 -- 172.21.15.78:0/1443526850 wait complete. 2024-09-15T20:43:02.542 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.507+0000 7f120b73c640 1 librados: shutdown 2024-09-15T20:43:02.542 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:43:02.542 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:43:02.542 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:43:02.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient: get_monmap_and_config 2024-09-15T20:43:02.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient: monmap: 2024-09-15T20:43:02.652 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.652 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.652 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:02.653524+0000 2024-09-15T20:43:02.652 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:02.653524+0000 2024-09-15T20:43:02.653 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.653 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.653 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.653 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:02.653 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:02.653 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 Processor -- start 2024-09-15T20:43:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- start start 2024-09-15T20:43:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient: init 2024-09-15T20:43:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:43:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0c135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0c13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0c0745e0 con 0x7f6a0c135b60 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0c06ecf0 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0c06efd0 con 0x7f6a0c13c220 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient(hunting): authenticate will time out at 1843.538086s 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0a59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6a0c135b60 0x7f6a0c135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55480/0 (socket says 172.21.15.78:55480) 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0a59c640 1 -- 172.21.15.78:0/920167443 learned_addr learned my addr 172.21.15.78:0/920167443 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 297292222 0 0) 0x7f6a0c0745e0 con 0x7f6a0c135b60 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient(hunting): my global_id is 114207 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69f4003530 con 0x7f6a0c135b60 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 310365840 0 0) 0x7f6a0c06efd0 con 0x7f6a0c13c220 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient(hunting): my global_id is 124103 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a0c0745e0 con 0x7f6a0c13c220 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4215766391 0 0) 0x7f69f4003530 con 0x7f6a0c135b60 2024-09-15T20:43:02.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 >> v1:172.21.15.78:6790/0 conn(0x7f6a0c13c220 legacy=0x7f6a0c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 >> v1:172.21.15.146:6789/0 conn(0x7f6a0c0a24e0 legacy=0x7f6a0c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 monclient: found mon.noname-a 2024-09-15T20:43:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0c136650 con 0x7f6a0c135b60 2024-09-15T20:43:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 5 monclient: authenticate success, global_id 114207 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6a04001a20 con 0x7f6a0c135b60 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: dump: 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.666 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a04001c40 con 0x7f6a0c135b60 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 4 set_mon_vals no callback set 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 1 -- 172.21.15.78:0/920167443 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6a04002050 con 0x7f6a0c135b60 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a0959a640 10 monclient: dump: 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 10 monclient: shutdown 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- 172.21.15.78:0/920167443 >> v1:172.21.15.78:6789/0 conn(0x7f6a0c135b60 legacy=0x7f6a0c135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- 172.21.15.78:0/920167443 shutdown_connections 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- 172.21.15.78:0/920167443 >> 172.21.15.78:0/920167443 conn(0x7f6a0c0a63c0 msgr2=0x7f6a0c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- 172.21.15.78:0/920167443 shutdown_connections 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.651+0000 7f6a10aaa640 1 -- 172.21.15.78:0/920167443 wait complete. 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: monmap: 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:02.669 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 librados: starting msgr at 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 librados: starting objecter 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 Processor -- start 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 -- start start 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 librados: setting wanted keys 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 librados: calling monclient init 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: init 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:43:02.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient(hunting): picked mon.noname-c con 0x7f6a0c0a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient(hunting): picked mon.noname-a con 0x7f6a0c135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient(hunting): picked mon.noname-b con 0x7f6a0c13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0c136910 con 0x7f6a0c135b60 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0c09d180 con 0x7f6a0c13c220 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6a0c138210 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient(hunting): authenticate will time out at 1843.540872s 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a0a59c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f6a0c0a24e0 0x7f6a0c071f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50846/0 (socket says 172.21.15.78:50846) 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a0a59c640 1 -- 172.21.15.78:0/1423610649 learned_addr learned my addr 172.21.15.78:0/1423610649 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.?.objecter ms_handle_connect 0x7f6a0c0a24e0 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.?.objecter ms_handle_connect 0x7f6a0c135b60 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 598837006 0 0) 0x7f6a0c138210 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): my global_id is 124106 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69ec004ec0 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655489880 0 0) 0x7f6a0c136910 con 0x7f6a0c135b60 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): my global_id is 114210 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a0c138210 con 0x7f6a0c135b60 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.?.objecter ms_handle_connect 0x7f6a0c13c220 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2322232394 0 0) 0x7f69ec004ec0 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6a0c136910 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6a04001f90 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:43:02.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): dump: 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3033649983 0 0) 0x7f6a0c138210 con 0x7f6a0c135b60 2024-09-15T20:43:02.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69ec004ec0 con 0x7f6a0c135b60 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6a00001880 con 0x7f6a0c135b60 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): dump: 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.675 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1060500466 0 0) 0x7f6a0c136910 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 >> v1:172.21.15.146:6789/0 conn(0x7f6a0c13c220 legacy=0x7f6a0c1d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 >> v1:172.21.15.78:6789/0 conn(0x7f6a0c135b60 legacy=0x7f6a0c072690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 monclient: found mon.c 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0c1384f0 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 5 monclient: authenticate success, global_id 124106 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: _renew_subs 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f6a0c136fd0 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 client.124106.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: _renew_subs 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f6a0c072da0 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 librados: init done 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6a04001ba0 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6a04005120 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:02.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a08d99640 4 set_mon_vals no callback set 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: dump: 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f69f00065d0 con 0x7f69f0002870 2024-09-15T20:43:02.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.124106.objecter ms_handle_connect 0x7f69f0002870 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.124106.objecter resend_mon_ops 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f6a04011190 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(445..445 src has 223..445) ==== 5354+0+0 (unknown 3722762973 0 0) 0x7f6a0404bbe0 con 0x7f6a0c0a24e0 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.124106.objecter ms_dispatch 0x7f6a0c000ba0 osd_map(445..445 src has 223..445) 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 3 client.124106.objecter handle_osd_map got epochs [445,445] > 0 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 3 client.124106.objecter handle_osd_map decoding full epoch 445 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 20 client.124106.objecter dump_active .. 0 homeless 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3430445670 0 0) 0x7f69f00065d0 con 0x7f69f0002870 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient(hunting): my global_id is 124109 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6a0c136910 con 0x7f69f0002870 2024-09-15T20:43:02.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1200029628 0 0) 0x7f6a0c136910 con 0x7f69f0002870 2024-09-15T20:43:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69f00065d0 con 0x7f69f0002870 2024-09-15T20:43:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6a00003990 con 0x7f69f0002870 2024-09-15T20:43:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: dump: 2024-09-15T20:43:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.680 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.681 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.682 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.682 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.682 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.682 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.682 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.682 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.682 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f6a0ad9d640 10 monclient: get_auth_request con 0x7f69ec03df90 auth_method 0 2024-09-15T20:43:02.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3585462535 0 0) 0x7f69f00065d0 con 0x7f69f0002870 2024-09-15T20:43:02.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69f000a170 con 0x7f69f0002870 2024-09-15T20:43:02.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 client.124106.objecter ms_handle_connect 0x7f69ec03df90 2024-09-15T20:43:02.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6a00005b70 con 0x7f69f0002870 2024-09-15T20:43:02.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:02.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 >> v1:172.21.15.78:6790/0 conn(0x7f69f0002870 legacy=0x7f69f0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.655+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 >> v1:172.21.15.78:6790/0 conn(0x7f69f0002870 legacy=0x7f69f0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:02.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:43:02.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f69f0009e60 con 0x7f69f0009120 2024-09-15T20:43:02.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 client.124106.objecter ms_handle_connect 0x7f69f0009120 2024-09-15T20:43:02.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 client.124106.objecter resend_mon_ops 2024-09-15T20:43:02.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 715100671 0 0) 0x7f69f0009e60 con 0x7f69f0009120 2024-09-15T20:43:02.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient(hunting): my global_id is 124112 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69f00065d0 con 0x7f69f0009120 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2218092163 0 0) 0x7f69f00065d0 con 0x7f69f0009120 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69f0009e60 con 0x7f69f0009120 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6a00004020 con 0x7f69f0009120 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient: dump: 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.687 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.688 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.688 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.688 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.688 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.688 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 771806305 0 0) 0x7f69f0009e60 con 0x7f69f0009120 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f69f000c400 con 0x7f69f0009120 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 1164548173) 0x7f6a000040e0 con 0x7f69f0009120 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 >> v1:172.21.15.78:6790/0 conn(0x7f69f0009120 legacy=0x7f69f00095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.689 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f69faffd640 1 -- 172.21.15.78:0/1423610649 >> v1:172.21.15.78:6790/0 conn(0x7f69f0009120 legacy=0x7f69f00095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:02.690 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:43:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:43:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 44, 2024-09-15T20:43:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 591, 2024-09-15T20:43:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:43:02.690 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.691 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:43:02.692 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.693 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:02.694 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.695 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.696 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.697 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.698 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.699 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.700 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.701 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:43:02.702 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:43:02.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 10 librados: watch_flush enter 2024-09-15T20:43:02.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 10 librados: watch_flush exit 2024-09-15T20:43:02.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 20 client.124106.objecter shutdown clearing up homeless session... 2024-09-15T20:43:02.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 10 client.124106.objecter successfully canceled tick 2024-09-15T20:43:02.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 >> v1:172.21.15.146:6812/435598642 conn(0x7f69ec03df90 legacy=0x7f69ec040450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 10 monclient: shutdown 2024-09-15T20:43:02.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 >> v1:172.21.15.78:6790/0 conn(0x7f6a0c0a24e0 legacy=0x7f6a0c071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a0ad9d640 1 -- 172.21.15.78:0/1423610649 reap_dead start 2024-09-15T20:43:02.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a0ad9d640 1 -- 172.21.15.78:0/1423610649 reap_dead start 2024-09-15T20:43:02.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 shutdown_connections 2024-09-15T20:43:02.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 >> 172.21.15.78:0/1423610649 conn(0x7f6a0c0a63c0 msgr2=0x7f6a0c12eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 shutdown_connections 2024-09-15T20:43:02.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 1 -- 172.21.15.78:0/1423610649 wait complete. 2024-09-15T20:43:02.705 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.663+0000 7f6a10aaa640 1 librados: shutdown 2024-09-15T20:43:02.705 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:43:02.705 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:43:02.706 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:43:02.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.807+0000 7f851add1640 10 monclient: get_monmap_and_config 2024-09-15T20:43:02.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.807+0000 7f851add1640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.807+0000 7f851add1640 10 monclient: monmap: 2024-09-15T20:43:02.810 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.810 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.810 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:02.811872+0000 2024-09-15T20:43:02.810 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:02.811872+0000 2024-09-15T20:43:02.811 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.811 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.811 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.811 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:02.811 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:02.811 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.807+0000 7f851add1640 1 Processor -- start 2024-09-15T20:43:02.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.807+0000 7f851add1640 1 -- start start 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.807+0000 7f851add1640 10 monclient: init 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): picked mon.noname-a con 0x7f8514134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): picked mon.noname-c con 0x7f851409f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): picked mon.noname-b con 0x7f851406e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85140a0470 con 0x7f8514134bb0 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f851409feb0 con 0x7f851409f5a0 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f851409e4a0 con 0x7f851406e960 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): authenticate will time out at 1843.696470s 2024-09-15T20:43:02.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f8518b6d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8514134bb0 0x7f8514134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55496/0 (socket says 172.21.15.78:55496) 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f8518b6d640 1 -- 172.21.15.78:0/3193485902 learned_addr learned my addr 172.21.15.78:0/3193485902 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851936e640 1 --1- 172.21.15.78:0/3193485902 >> v1:172.21.15.78:6790/0 conn(0x7f851406e960 0x7f851413d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50868/0 (socket says 172.21.15.78:50868) 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 855103531 0 0) 0x7f851409e4a0 con 0x7f851406e960 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient(hunting): my global_id is 124115 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84fc003530 con 0x7f851406e960 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1030516953 0 0) 0x7f85040014a0 con 0x7f8514134bb0 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient(hunting): my global_id is 114213 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f851409e4a0 con 0x7f8514134bb0 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2584000811 0 0) 0x7f851409e4a0 con 0x7f8514134bb0 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 >> v1:172.21.15.78:6790/0 conn(0x7f851406e960 legacy=0x7f851413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 >> v1:172.21.15.146:6789/0 conn(0x7f851409f5a0 legacy=0x7f85141399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 monclient: found mon.noname-a 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85140a0ff0 con 0x7f8514134bb0 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8504001bd0 con 0x7f8514134bb0 2024-09-15T20:43:02.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: dump: 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 5 monclient: authenticate success, global_id 114213 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8504001d70 con 0x7f8514134bb0 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 1 -- 172.21.15.78:0/3193485902 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8504004c30 con 0x7f8514134bb0 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 4 set_mon_vals no callback set 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85137fe640 10 monclient: dump: 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.824 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: shutdown 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- 172.21.15.78:0/3193485902 >> v1:172.21.15.78:6789/0 conn(0x7f8514134bb0 legacy=0x7f8514134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- 172.21.15.78:0/3193485902 shutdown_connections 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- 172.21.15.78:0/3193485902 >> 172.21.15.78:0/3193485902 conn(0x7f85140a63c0 msgr2=0x7f85140a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- 172.21.15.78:0/3193485902 shutdown_connections 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- 172.21.15.78:0/3193485902 wait complete. 2024-09-15T20:43:02.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: monmap: 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 librados: starting msgr at 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 librados: starting objecter 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 Processor -- start 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- start start 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 librados: setting wanted keys 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 librados: calling monclient init 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: init 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): picked mon.noname-c con 0x7f851406e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): picked mon.noname-a con 0x7f851409f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): picked mon.noname-b con 0x7f8514134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8514133730 con 0x7f851409f5a0 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8514133540 con 0x7f8514134bb0 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85141cd7b0 con 0x7f851406e960 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f851add1640 10 monclient(hunting): authenticate will time out at 1843.699258s 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f8518b6d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f851406e960 0x7f8514132e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50872/0 (socket says 172.21.15.78:50872) 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f8518b6d640 1 -- 172.21.15.78:0/849495131 learned_addr learned my addr 172.21.15.78:0/849495131 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 client.?.objecter ms_handle_connect 0x7f851406e960 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 client.?.objecter ms_handle_connect 0x7f851409f5a0 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2236325123 0 0) 0x7f85141cd7b0 con 0x7f851406e960 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): my global_id is 124118 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8500004ec0 con 0x7f851406e960 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1871964893 0 0) 0x7f8514133730 con 0x7f851409f5a0 2024-09-15T20:43:02.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): my global_id is 114216 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f85141cd7b0 con 0x7f851409f5a0 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 client.?.objecter ms_handle_connect 0x7f8514134bb0 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3607701034 0 0) 0x7f85141cd7b0 con 0x7f851409f5a0 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8514133730 con 0x7f851409f5a0 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8508001880 con 0x7f851409f5a0 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): dump: 2024-09-15T20:43:02.829 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1038309604 0 0) 0x7f8500004ec0 con 0x7f851406e960 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f85141cd7b0 con 0x7f851406e960 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3482426081 0 0) 0x7f8514133540 con 0x7f8514134bb0 2024-09-15T20:43:02.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): my global_id is 23692 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8500004ec0 con 0x7f8514134bb0 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8504001970 con 0x7f851406e960 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.811+0000 7f85117fa640 10 monclient(hunting): dump: 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.831 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3997203174 0 0) 0x7f8514133730 con 0x7f851409f5a0 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 >> v1:172.21.15.78:6790/0 conn(0x7f851406e960 legacy=0x7f8514132e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 >> v1:172.21.15.146:6789/0 conn(0x7f8514134bb0 legacy=0x7f85141d5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 monclient: found mon.a 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f851409d180 con 0x7f851409f5a0 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 5 monclient: authenticate success, global_id 114216 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 10 monclient: _renew_subs 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f85141224d0 con 0x7f851409f5a0 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 10 client.114216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 10 monclient: _renew_subs 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f8514123030 con 0x7f851409f5a0 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 1 librados: init done 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8508003600 con 0x7f851409f5a0 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8508003a10 con 0x7f851409f5a0 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f8512ffd640 4 set_mon_vals no callback set 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: dump: 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.834 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f84f40065d0 con 0x7f84f4002870 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f8508003c30 con 0x7f851409f5a0 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 client.114216.objecter ms_handle_connect 0x7f84f4002870 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 client.114216.objecter resend_mon_ops 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(445..445 src has 223..445) ==== 5354+0+0 (unknown 3722762973 0 0) 0x7f850804a5d0 con 0x7f851409f5a0 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 client.114216.objecter ms_dispatch 0x7f8514000ba0 osd_map(445..445 src has 223..445) 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 3 client.114216.objecter handle_osd_map got epochs [445,445] > 0 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 3 client.114216.objecter handle_osd_map decoding full epoch 445 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 20 client.114216.objecter dump_active .. 0 homeless 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 107967380 0 0) 0x7f84f40065d0 con 0x7f84f4002870 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient(hunting): my global_id is 23695 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8514133730 con 0x7f84f4002870 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f851936e640 10 monclient: get_auth_request con 0x7f850003e300 auth_method 0 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1330576226 0 0) 0x7f8514133730 con 0x7f84f4002870 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84f40065d0 con 0x7f84f4002870 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8504004ef0 con 0x7f84f4002870 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: dump: 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.836 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 client.114216.objecter ms_handle_connect 0x7f850003e300 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2547176375 0 0) 0x7f84f40065d0 con 0x7f84f4002870 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84f400a170 con 0x7f84f4002870 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8504007480 con 0x7f84f4002870 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:02.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 >> v1:172.21.15.146:6789/0 conn(0x7f84f4002870 legacy=0x7f84f4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.815+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 >> v1:172.21.15.146:6789/0 conn(0x7f84f4002870 legacy=0x7f84f4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:43:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f84f4009e80 con 0x7f84f4009140 2024-09-15T20:43:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 client.114216.objecter ms_handle_connect 0x7f84f4009140 2024-09-15T20:43:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 client.114216.objecter resend_mon_ops 2024-09-15T20:43:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1198260040 0 0) 0x7f84f4009e80 con 0x7f84f4009140 2024-09-15T20:43:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient(hunting): my global_id is 23698 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f84f40065d0 con 0x7f84f4009140 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4267485353 0 0) 0x7f84f40065d0 con 0x7f84f4009140 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f84f4009e80 con 0x7f84f4009140 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8504005220 con 0x7f84f4009140 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient: dump: 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.840 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1586978296 0 0) 0x7f84f4009e80 con 0x7f84f4009140 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f84f400c400 con 0x7f84f4009140 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 3940191797) 0x7f8504007630 con 0x7f84f4009140 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 >> v1:172.21.15.146:6789/0 conn(0x7f84f4009140 legacy=0x7f84f40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f85117fa640 1 -- 172.21.15.78:0/849495131 >> v1:172.21.15.146:6789/0 conn(0x7f84f4009140 legacy=0x7f84f40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:02.841 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 44, 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 695336, 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:43:02.842 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.843 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:43:02.844 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:02.845 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:02.846 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:43:02.847 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.848 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:43:02.850 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.851 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.852 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:43:02.853 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:43:02.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 10 librados: watch_flush enter 2024-09-15T20:43:02.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 10 librados: watch_flush exit 2024-09-15T20:43:02.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 20 client.114216.objecter shutdown clearing up homeless session... 2024-09-15T20:43:02.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 10 client.114216.objecter successfully canceled tick 2024-09-15T20:43:02.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 >> v1:172.21.15.146:6812/435598642 conn(0x7f850003e300 legacy=0x7f85000407c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 10 monclient: shutdown 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 >> v1:172.21.15.78:6789/0 conn(0x7f851409f5a0 legacy=0x7f85141d1c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851936e640 1 -- 172.21.15.78:0/849495131 reap_dead start 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851936e640 1 -- 172.21.15.78:0/849495131 reap_dead start 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 shutdown_connections 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.823+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 >> 172.21.15.78:0/849495131 conn(0x7f85140a63c0 msgr2=0x7f85140a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.827+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 shutdown_connections 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.827+0000 7f851add1640 1 -- 172.21.15.78:0/849495131 wait complete. 2024-09-15T20:43:02.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.827+0000 7f851add1640 1 librados: shutdown 2024-09-15T20:43:02.856 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:43:02.857 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:43:02.857 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:43:02.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.963+0000 7efd270e2640 10 monclient: get_monmap_and_config 2024-09-15T20:43:02.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.963+0000 7efd270e2640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.963+0000 7efd270e2640 10 monclient: monmap: 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:02.968235+0000 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:02.968235+0000 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:02.967 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 Processor -- start 2024-09-15T20:43:02.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- start start 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: init 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): picked mon.noname-c con 0x7efd20137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): picked mon.noname-b con 0x7efd20074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): picked mon.noname-a con 0x7efd2013e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd20138850 con 0x7efd2013e370 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd200a5da0 con 0x7efd20137d60 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd20073560 con 0x7efd20074500 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): authenticate will time out at 1843.852761s 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd2567f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7efd2013e370 0x7efd20140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55522/0 (socket says 172.21.15.78:55522) 2024-09-15T20:43:02.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd2567f640 1 -- 172.21.15.78:0/2495595155 learned_addr learned my addr 172.21.15.78:0/2495595155 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2354884554 0 0) 0x7efd20073560 con 0x7efd20074500 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient(hunting): my global_id is 124121 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd08003530 con 0x7efd20074500 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3802327168 0 0) 0x7efd20138850 con 0x7efd2013e370 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient(hunting): my global_id is 114219 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd20073560 con 0x7efd2013e370 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3414182980 0 0) 0x7efd08003530 con 0x7efd20074500 2024-09-15T20:43:02.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 >> v1:172.21.15.146:6789/0 conn(0x7efd20137d60 legacy=0x7efd20138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 >> v1:172.21.15.78:6789/0 conn(0x7efd2013e370 legacy=0x7efd20140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 monclient: found mon.noname-b 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd20073840 con 0x7efd20074500 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efd140016a0 con 0x7efd20074500 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: dump: 2024-09-15T20:43:02.976 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 5 monclient: authenticate success, global_id 124121 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd14001840 con 0x7efd20074500 2024-09-15T20:43:02.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 1 -- 172.21.15.78:0/2495595155 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efd14001c50 con 0x7efd20074500 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1f7fe640 10 monclient: dump: 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.978 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 4 set_mon_vals no callback set 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: shutdown 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- 172.21.15.78:0/2495595155 >> v1:172.21.15.78:6790/0 conn(0x7efd20074500 legacy=0x7efd2013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- 172.21.15.78:0/2495595155 shutdown_connections 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- 172.21.15.78:0/2495595155 >> 172.21.15.78:0/2495595155 conn(0x7efd200aa650 msgr2=0x7efd200aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- 172.21.15.78:0/2495595155 shutdown_connections 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- 172.21.15.78:0/2495595155 wait complete. 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: build_initial_monmap 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: monmap: 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:02.979 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 librados: starting msgr at 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 librados: starting objecter 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 Processor -- start 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- start start 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 librados: setting wanted keys 2024-09-15T20:43:02.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 librados: calling monclient init 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: init 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): picked mon.noname-c con 0x7efd20074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): picked mon.noname-a con 0x7efd20137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): picked mon.noname-b con 0x7efd2013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd2009e0b0 con 0x7efd20137d60 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd20141810 con 0x7efd2013e370 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd201325d0 con 0x7efd20074500 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd270e2640 10 monclient(hunting): authenticate will time out at 1843.855729s 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd24e7e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7efd20074500 0x7efd2009e780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50892/0 (socket says 172.21.15.78:50892) 2024-09-15T20:43:02.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd24e7e640 1 -- 172.21.15.78:0/4070174985 learned_addr learned my addr 172.21.15.78:0/4070174985 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1ffff640 1 --1- 172.21.15.78:0/4070174985 >> v1:172.21.15.78:6789/0 conn(0x7efd20137d60 0x7efd2009ee90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55538/0 (socket says 172.21.15.78:55538) 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1d7fa640 10 client.?.objecter ms_handle_connect 0x7efd20074500 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1d7fa640 10 client.?.objecter ms_handle_connect 0x7efd20137d60 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.967+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4090834741 0 0) 0x7efd2009e0b0 con 0x7efd20137d60 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): my global_id is 114222 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd0c004ec0 con 0x7efd20137d60 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2579083072 0 0) 0x7efd201325d0 con 0x7efd20074500 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): my global_id is 124124 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd2009e0b0 con 0x7efd20074500 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 client.?.objecter ms_handle_connect 0x7efd2013e370 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2158221545 0 0) 0x7efd0c004ec0 con 0x7efd20137d60 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd201325d0 con 0x7efd20137d60 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efd14001bd0 con 0x7efd20137d60 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): dump: 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.983 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 71065688 0 0) 0x7efd2009e0b0 con 0x7efd20074500 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7efd0c004ec0 con 0x7efd20074500 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efd100019c0 con 0x7efd20074500 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:02.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): dump: 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 44554505 0 0) 0x7efd20141810 con 0x7efd2013e370 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:02.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient(hunting): my global_id is 23704 2024-09-15T20:43:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efd2009e0b0 con 0x7efd2013e370 2024-09-15T20:43:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1729942290 0 0) 0x7efd201325d0 con 0x7efd20137d60 2024-09-15T20:43:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 >> v1:172.21.15.78:6790/0 conn(0x7efd20074500 legacy=0x7efd2009e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 >> v1:172.21.15.146:6789/0 conn(0x7efd2013e370 legacy=0x7efd2009f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:43:02.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 monclient: found mon.a 2024-09-15T20:43:02.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd201328b0 con 0x7efd20137d60 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient: _finish_auth 0 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 5 monclient: authenticate success, global_id 114222 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 10 monclient: _renew_subs 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7efd200a0b70 con 0x7efd20137d60 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 10 client.114222.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 10 monclient: _renew_subs 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7efd201d5e10 con 0x7efd20137d60 2024-09-15T20:43:02.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 1 librados: init done 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7efd14001820 con 0x7efd20137d60 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7efd14003b10 con 0x7efd20137d60 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1effd640 4 set_mon_vals no callback set 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 monclient: dump: 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:02.989 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd00005090 con 0x7efd20137d60 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7efd14003cb0 con 0x7efd20137d60 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(445..445 src has 223..445) ==== 5354+0+0 (unknown 3722762973 0 0) 0x7efd1404b670 con 0x7efd20137d60 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 client.114222.objecter ms_dispatch 0x7efd20000ba0 osd_map(445..445 src has 223..445) 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 3 client.114222.objecter handle_osd_map got epochs [445,445] > 0 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 3 client.114222.objecter handle_osd_map decoding full epoch 445 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 20 client.114222.objecter dump_active .. 0 homeless 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd24e7e640 10 monclient: get_auth_request con 0x7efd0c03e1a0 auth_method 0 2024-09-15T20:43:02.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.971+0000 7efd1d7fa640 10 client.114222.objecter ms_handle_connect 0x7efd0c03e1a0 2024-09-15T20:43:02.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.975+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7efd14016a50 con 0x7efd20137d60 2024-09-15T20:43:02.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.975+0000 7efd1d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:02.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:02.975+0000 7efd1d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:03.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.199+0000 7efd270e2640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:43:03.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.199+0000 7efd270e2640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:43:03.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.199+0000 7efd270e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:03.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.199+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7efd00005c50 con 0x7efd20137d60 2024-09-15T20:43:03.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.199+0000 7efd1d7fa640 1 -- 172.21.15.78:0/4070174985 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7efd1401add0 con 0x7efd20137d60 2024-09-15T20:43:03.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.199+0000 7efd1d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:43:03.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.199+0000 7efd1d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:03.203 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.203 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=9,logm=60} crc {auth=3095440526,config=2586702516,health=2791171403,kv=319579803,logm=3783589430}) 2024-09-15T20:43:03.203 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=10,kv=12,logm=56} crc {auth=1796689562,config=2586702516,health=1870099501,kv=199072906,logm=111023098}) 2024-09-15T20:43:03.203 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.204 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=17,config=2,health=12,kv=9,logm=60} crc {auth=3095440526,config=2586702516,health=2791171403,kv=319579803,logm=3783589430}) 2024-09-15T20:43:03.204 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=20,config=2,health=11,kv=10,logm=57} crc {auth=1796689562,config=575665002,health=867127297,kv=2609990224,logm=4080090282}) 2024-09-15T20:43:03.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 10 librados: watch_flush enter 2024-09-15T20:43:03.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 10 librados: watch_flush exit 2024-09-15T20:43:03.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 20 client.114222.objecter shutdown clearing up homeless session... 2024-09-15T20:43:03.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 10 client.114222.objecter successfully canceled tick 2024-09-15T20:43:03.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 >> v1:172.21.15.146:6812/435598642 conn(0x7efd0c03e1a0 legacy=0x7efd0c040660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:03.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 10 monclient: shutdown 2024-09-15T20:43:03.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:03.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 >> v1:172.21.15.78:6789/0 conn(0x7efd20137d60 legacy=0x7efd2009ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:03.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 shutdown_connections 2024-09-15T20:43:03.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 >> 172.21.15.78:0/4070174985 conn(0x7efd200aa650 msgr2=0x7efd200a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:03.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 shutdown_connections 2024-09-15T20:43:03.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 1 -- 172.21.15.78:0/4070174985 wait complete. 2024-09-15T20:43:03.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:03.203+0000 7efd270e2640 1 librados: shutdown 2024-09-15T20:43:03.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220059636}) 2024-09-15T20:43:03.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1328086426}) 2024-09-15T20:43:03.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220059636}) 2024-09-15T20:43:03.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.203+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3155029}) 2024-09-15T20:43:03.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491341470}) 2024-09-15T20:43:03.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=171551212}) 2024-09-15T20:43:03.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=491341470}) 2024-09-15T20:43:03.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1181162124}) 2024-09-15T20:43:03.210 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.210 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365948130}) 2024-09-15T20:43:03.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619134318}) 2024-09-15T20:43:03.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3365948130}) 2024-09-15T20:43:03.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.207+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=895300284}) 2024-09-15T20:43:03.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068231295}) 2024-09-15T20:43:03.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2258961940}) 2024-09-15T20:43:03.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3068231295}) 2024-09-15T20:43:03.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4169400075}) 2024-09-15T20:43:03.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752620300}) 2024-09-15T20:43:03.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2051750919}) 2024-09-15T20:43:03.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=752620300}) 2024-09-15T20:43:03.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.211+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1050071806}) 2024-09-15T20:43:03.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560496690}) 2024-09-15T20:43:03.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=210259955}) 2024-09-15T20:43:03.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2560496690}) 2024-09-15T20:43:03.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3672062473}) 2024-09-15T20:43:03.219 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.219 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056926375}) 2024-09-15T20:43:03.219 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3031333622}) 2024-09-15T20:43:03.219 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.219 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1056926375}) 2024-09-15T20:43:03.219 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.215+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3871966371}) 2024-09-15T20:43:03.220 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:43:03.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1793129522}) 2024-09-15T20:43:03.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2967677924}) 2024-09-15T20:43:03.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1793129522}) 2024-09-15T20:43:03.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=357537169}) 2024-09-15T20:43:03.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1969489468}) 2024-09-15T20:43:03.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=967553868}) 2024-09-15T20:43:03.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1969489468}) 2024-09-15T20:43:03.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.219+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1964731012}) 2024-09-15T20:43:03.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196547897}) 2024-09-15T20:43:03.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3131539596}) 2024-09-15T20:43:03.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1196547897}) 2024-09-15T20:43:03.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3153160927}) 2024-09-15T20:43:03.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390400688}) 2024-09-15T20:43:03.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3177311854}) 2024-09-15T20:43:03.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2390400688}) 2024-09-15T20:43:03.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.223+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2080609404}) 2024-09-15T20:43:03.229 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.229 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202629587}) 2024-09-15T20:43:03.229 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=78060694}) 2024-09-15T20:43:03.229 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.229 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3202629587}) 2024-09-15T20:43:03.229 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1177834560}) 2024-09-15T20:43:03.231 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.231 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996626694}) 2024-09-15T20:43:03.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.227+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=626713841}) 2024-09-15T20:43:03.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1996626694}) 2024-09-15T20:43:03.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3606789177}) 2024-09-15T20:43:03.233 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.233 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527898011}) 2024-09-15T20:43:03.233 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=777409584}) 2024-09-15T20:43:03.233 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.234 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1527898011}) 2024-09-15T20:43:03.234 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.231+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=663570954}) 2024-09-15T20:43:03.235 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=223522549}) 2024-09-15T20:43:03.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3962442276}) 2024-09-15T20:43:03.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=223522549}) 2024-09-15T20:43:03.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=549377943}) 2024-09-15T20:43:03.237 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.237 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003449161}) 2024-09-15T20:43:03.237 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1868918198}) 2024-09-15T20:43:03.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1003449161}) 2024-09-15T20:43:03.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.235+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1176729455}) 2024-09-15T20:43:03.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3987356782}) 2024-09-15T20:43:03.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3525225194}) 2024-09-15T20:43:03.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3987356782}) 2024-09-15T20:43:03.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1442057271}) 2024-09-15T20:43:03.241 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.241 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509920652}) 2024-09-15T20:43:03.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3441089705}) 2024-09-15T20:43:03.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=509920652}) 2024-09-15T20:43:03.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.239+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1712909142}) 2024-09-15T20:43:03.244 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.244 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358493000}) 2024-09-15T20:43:03.244 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2587598539}) 2024-09-15T20:43:03.244 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.244 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1358493000}) 2024-09-15T20:43:03.244 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4067369775}) 2024-09-15T20:43:03.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=233834657}) 2024-09-15T20:43:03.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3106767045}) 2024-09-15T20:43:03.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=233834657}) 2024-09-15T20:43:03.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.243+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1199905585}) 2024-09-15T20:43:03.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884124838}) 2024-09-15T20:43:03.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3373746441}) 2024-09-15T20:43:03.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884124838}) 2024-09-15T20:43:03.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1328323011}) 2024-09-15T20:43:03.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269505580}) 2024-09-15T20:43:03.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=244819561}) 2024-09-15T20:43:03.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4269505580}) 2024-09-15T20:43:03.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.247+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2397716248}) 2024-09-15T20:43:03.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367256849}) 2024-09-15T20:43:03.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3565915774}) 2024-09-15T20:43:03.253 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.253 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2367256849}) 2024-09-15T20:43:03.253 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1040083037}) 2024-09-15T20:43:03.254 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622437883}) 2024-09-15T20:43:03.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3864246743}) 2024-09-15T20:43:03.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622437883}) 2024-09-15T20:43:03.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.251+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1437509052}) 2024-09-15T20:43:03.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149195609}) 2024-09-15T20:43:03.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=983893444}) 2024-09-15T20:43:03.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4149195609}) 2024-09-15T20:43:03.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2687784621}) 2024-09-15T20:43:03.259 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.259 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3066348513}) 2024-09-15T20:43:03.259 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=372566789}) 2024-09-15T20:43:03.259 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.259 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3066348513}) 2024-09-15T20:43:03.259 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.255+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=659064234}) 2024-09-15T20:43:03.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.261 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=489364590}) 2024-09-15T20:43:03.261 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1880348595}) 2024-09-15T20:43:03.261 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.261 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=489364590}) 2024-09-15T20:43:03.261 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2168791153}) 2024-09-15T20:43:03.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634466492}) 2024-09-15T20:43:03.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1619292871}) 2024-09-15T20:43:03.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2634466492}) 2024-09-15T20:43:03.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.259+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1265609754}) 2024-09-15T20:43:03.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231624979}) 2024-09-15T20:43:03.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2380851408}) 2024-09-15T20:43:03.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231624979}) 2024-09-15T20:43:03.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=777843078}) 2024-09-15T20:43:03.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46243361}) 2024-09-15T20:43:03.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.263+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3373997471}) 2024-09-15T20:43:03.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=46243361}) 2024-09-15T20:43:03.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3345016854}) 2024-09-15T20:43:03.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272466362}) 2024-09-15T20:43:03.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782834743}) 2024-09-15T20:43:03.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=272466362}) 2024-09-15T20:43:03.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1946078640}) 2024-09-15T20:43:03.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.267+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289058760}) 2024-09-15T20:43:03.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=283983944}) 2024-09-15T20:43:03.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1289058760}) 2024-09-15T20:43:03.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3748618814}) 2024-09-15T20:43:03.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505418825}) 2024-09-15T20:43:03.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2579806112}) 2024-09-15T20:43:03.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=505418825}) 2024-09-15T20:43:03.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1245976547}) 2024-09-15T20:43:03.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=679429250}) 2024-09-15T20:43:03.276 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3037151157}) 2024-09-15T20:43:03.276 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.276 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=679429250}) 2024-09-15T20:43:03.276 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.271+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4172721496}) 2024-09-15T20:43:03.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967018482}) 2024-09-15T20:43:03.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3723123367}) 2024-09-15T20:43:03.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=967018482}) 2024-09-15T20:43:03.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3517993141}) 2024-09-15T20:43:03.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3569667675}) 2024-09-15T20:43:03.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2561805639}) 2024-09-15T20:43:03.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3569667675}) 2024-09-15T20:43:03.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.275+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=901199193}) 2024-09-15T20:43:03.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389515911}) 2024-09-15T20:43:03.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=258109339}) 2024-09-15T20:43:03.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1389515911}) 2024-09-15T20:43:03.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=790115432}) 2024-09-15T20:43:03.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477792196}) 2024-09-15T20:43:03.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2991190020}) 2024-09-15T20:43:03.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477792196}) 2024-09-15T20:43:03.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.279+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2026875207}) 2024-09-15T20:43:03.285 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.285 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763274449}) 2024-09-15T20:43:03.285 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3068782711}) 2024-09-15T20:43:03.285 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.285 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3763274449}) 2024-09-15T20:43:03.285 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1118289619}) 2024-09-15T20:43:03.287 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.287 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008124112}) 2024-09-15T20:43:03.287 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3301579391}) 2024-09-15T20:43:03.287 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.287 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1008124112}) 2024-09-15T20:43:03.287 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.283+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1782024415}) 2024-09-15T20:43:03.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135481055}) 2024-09-15T20:43:03.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=101107490}) 2024-09-15T20:43:03.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=135481055}) 2024-09-15T20:43:03.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2614509250}) 2024-09-15T20:43:03.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435242853}) 2024-09-15T20:43:03.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3675793361}) 2024-09-15T20:43:03.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2435242853}) 2024-09-15T20:43:03.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.287+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3585212674}) 2024-09-15T20:43:03.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193179266}) 2024-09-15T20:43:03.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1239984065}) 2024-09-15T20:43:03.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1193179266}) 2024-09-15T20:43:03.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1512932267}) 2024-09-15T20:43:03.295 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.295 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886483688}) 2024-09-15T20:43:03.295 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2831245929}) 2024-09-15T20:43:03.295 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.295 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1886483688}) 2024-09-15T20:43:03.295 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.291+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808722450}) 2024-09-15T20:43:03.297 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.297 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969166213}) 2024-09-15T20:43:03.297 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3699185358}) 2024-09-15T20:43:03.297 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.297 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3969166213}) 2024-09-15T20:43:03.298 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4244179258}) 2024-09-15T20:43:03.299 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.299 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119104659}) 2024-09-15T20:43:03.299 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2725655049}) 2024-09-15T20:43:03.299 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.299 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3119104659}) 2024-09-15T20:43:03.299 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.295+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2297655451}) 2024-09-15T20:43:03.302 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.299+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.302 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.299+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=13035914,mdsmap=1463177238,mgr=2271130127,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=3355335443}) 2024-09-15T20:43:03.302 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.299+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=82,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=5} crc {logm=499829933,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488952,mgrstat=470690179}) 2024-09-15T20:43:03.302 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.299+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.302 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.299+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=78,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=13035914,mdsmap=1463177238,mgr=2271130127,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=3355335443}) 2024-09-15T20:43:03.302 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.299+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=78,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10} crc {logm=2443045073,mdsmap=503983409,mgr=2271130127,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=3355335443}) 2024-09-15T20:43:03.304 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.304 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=39} crc {mgrstat=3040917445,mon_config_key=342708592,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1681388020,osd_pg_creating=1725184239,osd_snap=1756326332,osdmap=1193469407}) 2024-09-15T20:43:03.304 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mgrstat=2,mon_config_key=17,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=35} crc {mgrstat=210638404,mon_config_key=3551184854,monmap=3202364208,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=1725184239,osd_snap=3446850117,osdmap=2936749239}) 2024-09-15T20:43:03.304 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.304 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mgrstat=1,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=32,osdmap=39} crc {mgrstat=3040917445,mon_config_key=342708592,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1681388020,osd_pg_creating=1725184239,osd_snap=1756326332,osdmap=1193469407}) 2024-09-15T20:43:03.304 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mgrstat=2,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=33,osdmap=36} crc {mgrstat=210638404,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=598337686,osd_metadata=159488793,osd_pg_creating=1725184239,osd_snap=2313100086,osdmap=349507771}) 2024-09-15T20:43:03.306 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.306 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1395659389}) 2024-09-15T20:43:03.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=489175041}) 2024-09-15T20:43:03.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1395659389}) 2024-09-15T20:43:03.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.303+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3824429037}) 2024-09-15T20:43:03.309 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.307+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.310 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.307+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2818113907}) 2024-09-15T20:43:03.310 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.307+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=4236039722}) 2024-09-15T20:43:03.310 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.307+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.310 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.307+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2818113907}) 2024-09-15T20:43:03.310 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.307+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=2797953291}) 2024-09-15T20:43:03.313 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.311+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.313 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.311+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=3590321424}) 2024-09-15T20:43:03.313 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.311+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=96} crc {osdmap=1595100090}) 2024-09-15T20:43:03.314 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.311+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:43:03.314 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.311+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=93} crc {osdmap=3590321424}) 2024-09-15T20:43:03.314 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:43:03.311+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=90} crc {osdmap=2914406538}) 2024-09-15T20:43:04.221 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:43:04.222 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:43:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient: get_monmap_and_config 2024-09-15T20:43:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient: build_initial_monmap 2024-09-15T20:43:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient: monmap: 2024-09-15T20:43:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:04.345 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:04.346669+0000 2024-09-15T20:43:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:04.346669+0000 2024-09-15T20:43:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:04.346 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:04.346 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 1 Processor -- start 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 1 -- start start 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient: init 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient(hunting): picked mon.noname-c con 0x7f5788074630 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient(hunting): picked mon.noname-a con 0x7f578809d950 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient(hunting): picked mon.noname-b con 0x7f5788074a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5788134bc0 con 0x7f578809d950 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578809f900 con 0x7f5788074630 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578809fbe0 con 0x7f5788074a10 2024-09-15T20:43:04.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.343+0000 7f578f542640 10 monclient(hunting): authenticate will time out at 1845.231968s 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578cadd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f578809d950 0x7f57881344b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55552/0 (socket says 172.21.15.78:55552) 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578cadd640 1 -- 172.21.15.78:0/1113814977 learned_addr learned my addr 172.21.15.78:0/1113814977 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4130112285 0 0) 0x7f578809fbe0 con 0x7f5788074a10 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient(hunting): my global_id is 124127 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5774003530 con 0x7f5788074a10 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 992619383 0 0) 0x7f5788134bc0 con 0x7f578809d950 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient(hunting): my global_id is 114225 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f578809fbe0 con 0x7f578809d950 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4093698107 0 0) 0x7f5774003530 con 0x7f5788074a10 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 >> v1:172.21.15.146:6789/0 conn(0x7f5788074630 legacy=0x7f5788133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 >> v1:172.21.15.78:6789/0 conn(0x7f578809d950 legacy=0x7f57881344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 monclient: found mon.noname-b 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f578809fec0 con 0x7f5788074a10 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: _finish_auth 0 2024-09-15T20:43:04.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5780001a20 con 0x7f5788074a10 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 5 monclient: authenticate success, global_id 124127 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: dump: 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.357 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5780001bc0 con 0x7f5788074a10 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 4 set_mon_vals no callback set 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 1 -- 172.21.15.78:0/1113814977 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5780001fd0 con 0x7f5788074a10 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577bfff640 10 monclient: dump: 2024-09-15T20:43:04.358 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: shutdown 2024-09-15T20:43:04.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- 172.21.15.78:0/1113814977 >> v1:172.21.15.78:6790/0 conn(0x7f5788074a10 legacy=0x7f578813d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- 172.21.15.78:0/1113814977 shutdown_connections 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- 172.21.15.78:0/1113814977 >> 172.21.15.78:0/1113814977 conn(0x7f57880a63c0 msgr2=0x7f57880a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- 172.21.15.78:0/1113814977 shutdown_connections 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- 172.21.15.78:0/1113814977 wait complete. 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: build_initial_monmap 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: monmap: 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.360 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 librados: starting msgr at 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 librados: starting objecter 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 Processor -- start 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- start start 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 librados: setting wanted keys 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 librados: calling monclient init 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: init 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:04.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient(hunting): picked mon.noname-b con 0x7f5788074630 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient(hunting): picked mon.noname-a con 0x7f5788074a10 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient(hunting): picked mon.noname-c con 0x7f578809d950 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578809d180 con 0x7f5788074a10 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f57881c8910 con 0x7f5788074630 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f578806fe50 con 0x7f578809d950 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient(hunting): authenticate will time out at 1845.234899s 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578cadd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5788074a10 0x7f57881cd550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55562/0 (socket says 172.21.15.78:55562) 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578cadd640 1 -- 172.21.15.78:0/633341370 learned_addr learned my addr 172.21.15.78:0/633341370 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 client.?.objecter ms_handle_connect 0x7f5788074a10 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 client.?.objecter ms_handle_connect 0x7f578809d950 2024-09-15T20:43:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3678725781 0 0) 0x7f578809d180 con 0x7f5788074a10 2024-09-15T20:43:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): my global_id is 114228 2024-09-15T20:43:04.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f576c004ef0 con 0x7f5788074a10 2024-09-15T20:43:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 524035774 0 0) 0x7f578806fe50 con 0x7f578809d950 2024-09-15T20:43:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): my global_id is 124130 2024-09-15T20:43:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f578809d180 con 0x7f578809d950 2024-09-15T20:43:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 client.?.objecter ms_handle_connect 0x7f5788074630 2024-09-15T20:43:04.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3653315416 0 0) 0x7f576c004ef0 con 0x7f5788074a10 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f578806fe50 con 0x7f5788074a10 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1157687150 0 0) 0x7f578809d180 con 0x7f578809d950 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f576c004ef0 con 0x7f578809d950 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f577c0013a0 con 0x7f5788074a10 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): dump: 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.365 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5780001f10 con 0x7f578809d950 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): dump: 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.366 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2975454275 0 0) 0x7f57881c8910 con 0x7f5788074630 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient(hunting): my global_id is 23710 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f578809d180 con 0x7f5788074630 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3958857818 0 0) 0x7f578806fe50 con 0x7f5788074a10 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 >> v1:172.21.15.78:6790/0 conn(0x7f578809d950 legacy=0x7f57881d0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 >> v1:172.21.15.146:6789/0 conn(0x7f5788074630 legacy=0x7f57880736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 monclient: found mon.a 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57880721b0 con 0x7f5788074a10 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient: _finish_auth 0 2024-09-15T20:43:04.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 5 monclient: authenticate success, global_id 114228 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: _renew_subs 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5788071910 con 0x7f5788074a10 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 client.114228.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: _renew_subs 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5788073fc0 con 0x7f5788074a10 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f578f542640 1 librados: init done 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f577c0016a0 con 0x7f5788074a10 2024-09-15T20:43:04.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f577c004c40 con 0x7f5788074a10 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f577b7fe640 4 set_mon_vals no callback set 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.347+0000 7f5779ffb640 10 monclient: dump: 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.370 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f578f542640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f578f542640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f578f542640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f577c010cb0 con 0x7f5788074a10 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5768005090 con 0x7f5788074a10 2024-09-15T20:43:04.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(447..447 src has 223..447) ==== 5354+0+0 (unknown 3060371923 0 0) 0x7f577c04b740 con 0x7f5788074a10 2024-09-15T20:43:04.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f5779ffb640 10 client.114228.objecter ms_dispatch 0x7f5788000ba0 osd_map(447..447 src has 223..447) 2024-09-15T20:43:04.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f5779ffb640 3 client.114228.objecter handle_osd_map got epochs [447,447] > 0 2024-09-15T20:43:04.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f5779ffb640 3 client.114228.objecter handle_osd_map decoding full epoch 447 2024-09-15T20:43:04.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f5779ffb640 20 client.114228.objecter dump_active .. 0 homeless 2024-09-15T20:43:04.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f578d2de640 10 monclient: get_auth_request con 0x7f576c03e2a0 auth_method 0 2024-09-15T20:43:04.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.351+0000 7f5779ffb640 10 client.114228.objecter ms_handle_connect 0x7f576c03e2a0 2024-09-15T20:43:04.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.355+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f577c016ac0 con 0x7f5788074a10 2024-09-15T20:43:04.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.355+0000 7f5779ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:04.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.355+0000 7f5779ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:04.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:43:04.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:04.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:04.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5768005380 con 0x7f5788074a10 2024-09-15T20:43:04.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f5779ffb640 1 -- 172.21.15.78:0/633341370 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 508567491) 0x7f577c01aea0 con 0x7f5788074a10 2024-09-15T20:43:04.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f5779ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:04.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f5779ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:04.574 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:43:04.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 10 librados: watch_flush enter 2024-09-15T20:43:04.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 10 librados: watch_flush exit 2024-09-15T20:43:04.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 20 client.114228.objecter shutdown clearing up homeless session... 2024-09-15T20:43:04.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 10 client.114228.objecter successfully canceled tick 2024-09-15T20:43:04.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 >> v1:172.21.15.146:6812/435598642 conn(0x7f576c03e2a0 legacy=0x7f576c040760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 10 monclient: shutdown 2024-09-15T20:43:04.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:04.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 >> v1:172.21.15.78:6789/0 conn(0x7f5788074a10 legacy=0x7f57881cd550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 shutdown_connections 2024-09-15T20:43:04.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 >> 172.21.15.78:0/633341370 conn(0x7f57880a63c0 msgr2=0x7f57880a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:04.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 shutdown_connections 2024-09-15T20:43:04.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 1 -- 172.21.15.78:0/633341370 wait complete. 2024-09-15T20:43:04.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.571+0000 7f578f542640 1 librados: shutdown 2024-09-15T20:43:04.590 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:43:04.590 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:43:04.590 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:43:04.590 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:43:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient: get_monmap_and_config 2024-09-15T20:43:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient: build_initial_monmap 2024-09-15T20:43:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient: monmap: 2024-09-15T20:43:04.704 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:04.705 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.705 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:04.705762+0000 2024-09-15T20:43:04.705 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:04.705762+0000 2024-09-15T20:43:04.705 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:04.705 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.706 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:04.706 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:04.706 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:04.706 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 Processor -- start 2024-09-15T20:43:04.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- start start 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient: init 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient(hunting): picked mon.noname-a con 0x7ff1d4135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient(hunting): picked mon.noname-b con 0x7ff1d40a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient(hunting): picked mon.noname-c con 0x7ff1d413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d40745e0 con 0x7ff1d4135b60 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d406ecf0 con 0x7ff1d413c220 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d406efd0 con 0x7ff1d40a24e0 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient(hunting): authenticate will time out at 1845.590282s 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1d959c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff1d4135b60 0x7ff1d4135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55576/0 (socket says 172.21.15.78:55576) 2024-09-15T20:43:04.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1d959c640 1 -- 172.21.15.78:0/2636356765 learned_addr learned my addr 172.21.15.78:0/2636356765 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3181519490 0 0) 0x7ff1d40745e0 con 0x7ff1d4135b60 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient(hunting): my global_id is 114231 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1c0003530 con 0x7ff1d4135b60 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1686179333 0 0) 0x7ff1d406efd0 con 0x7ff1d40a24e0 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient(hunting): my global_id is 124133 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1d40745e0 con 0x7ff1d40a24e0 2024-09-15T20:43:04.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3177731670 0 0) 0x7ff1c0003530 con 0x7ff1d4135b60 2024-09-15T20:43:04.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 >> v1:172.21.15.78:6790/0 conn(0x7ff1d40a24e0 legacy=0x7ff1d413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 >> v1:172.21.15.146:6789/0 conn(0x7ff1d413c220 legacy=0x7ff1d413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: _finish_hunting 0 2024-09-15T20:43:04.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 monclient: found mon.noname-a 2024-09-15T20:43:04.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:04.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:43:04.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d4136650 con 0x7ff1d4135b60 2024-09-15T20:43:04.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: _finish_auth 0 2024-09-15T20:43:04.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: _check_auth_tickets 2024-09-15T20:43:04.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:04.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1c40016a0 con 0x7ff1d4135b60 2024-09-15T20:43:04.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:04.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: dump: 2024-09-15T20:43:04.718 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.718 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.718 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.718 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.718 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.718 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.718 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.718 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 5 monclient: authenticate success, global_id 114231 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1c40018c0 con 0x7ff1d4135b60 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 1 -- 172.21.15.78:0/2636356765 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1c4001cd0 con 0x7ff1d4135b60 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 4 set_mon_vals no callback set 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1cbfff640 10 monclient: dump: 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.719 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 10 monclient: shutdown 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- 172.21.15.78:0/2636356765 >> v1:172.21.15.78:6789/0 conn(0x7ff1d4135b60 legacy=0x7ff1d4135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- 172.21.15.78:0/2636356765 shutdown_connections 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- 172.21.15.78:0/2636356765 >> 172.21.15.78:0/2636356765 conn(0x7ff1d40a63c0 msgr2=0x7ff1d40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- 172.21.15.78:0/2636356765 shutdown_connections 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.703+0000 7ff1db800640 1 -- 172.21.15.78:0/2636356765 wait complete. 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: build_initial_monmap 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: monmap: 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:04.721 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 librados: starting msgr at 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 librados: starting objecter 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 Processor -- start 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 -- start start 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 librados: setting wanted keys 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 librados: calling monclient init 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: init 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:43:04.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient(hunting): picked mon.noname-a con 0x7ff1d40a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient(hunting): picked mon.noname-c con 0x7ff1d4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient(hunting): picked mon.noname-b con 0x7ff1d413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d413b600 con 0x7ff1d40a24e0 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d40735b0 con 0x7ff1d413c220 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1d409d180 con 0x7ff1d4135b60 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient(hunting): authenticate will time out at 1845.593458s 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1d959c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff1d40a24e0 0x7ff1d4071d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55588/0 (socket says 172.21.15.78:55588) 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1d959c640 1 -- 172.21.15.78:0/3900816455 learned_addr learned my addr 172.21.15.78:0/3900816455 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.?.objecter ms_handle_connect 0x7ff1d40a24e0 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.?.objecter ms_handle_connect 0x7ff1d4135b60 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1445919339 0 0) 0x7ff1d413b600 con 0x7ff1d40a24e0 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): my global_id is 114234 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b8004ec0 con 0x7ff1d40a24e0 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 732712663 0 0) 0x7ff1d409d180 con 0x7ff1d4135b60 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): my global_id is 124136 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1d413b600 con 0x7ff1d4135b60 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4235739211 0 0) 0x7ff1d413b600 con 0x7ff1d4135b60 2024-09-15T20:43:04.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1d409d180 con 0x7ff1d4135b60 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.?.objecter ms_handle_connect 0x7ff1d413c220 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1d00012e0 con 0x7ff1d4135b60 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): dump: 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.725 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1969009582 0 0) 0x7ff1b8004ec0 con 0x7ff1d40a24e0 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1d413b600 con 0x7ff1d40a24e0 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1c4001c10 con 0x7ff1d40a24e0 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): dump: 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.726 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3541863968 0 0) 0x7ff1d409d180 con 0x7ff1d4135b60 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 >> v1:172.21.15.146:6789/0 conn(0x7ff1d413c220 legacy=0x7ff1d41d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 >> v1:172.21.15.78:6789/0 conn(0x7ff1d40a24e0 legacy=0x7ff1d4071d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 monclient: found mon.c 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:04.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1d4138600 con 0x7ff1d4135b60 2024-09-15T20:43:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:43:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:43:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 5 monclient: authenticate success, global_id 124136 2024-09-15T20:43:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: _renew_subs 2024-09-15T20:43:04.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff1d4135040 con 0x7ff1d4135b60 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 client.124136.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: _renew_subs 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff1d4137810 con 0x7ff1d4135b60 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 librados: init done 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff1d0003600 con 0x7ff1d4135b60 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1d0003950 con 0x7ff1d4135b60 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1cb7fe640 4 set_mon_vals no callback set 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: dump: 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.730 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff1b40065d0 con 0x7ff1b4002870 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff1d000f9c0 con 0x7ff1d4135b60 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.124136.objecter ms_handle_connect 0x7ff1b4002870 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.124136.objecter resend_mon_ops 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(447..447 src has 223..447) ==== 5354+0+0 (unknown 3060371923 0 0) 0x7ff1d004a600 con 0x7ff1d4135b60 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 client.124136.objecter ms_dispatch 0x7ff1d4000ba0 osd_map(447..447 src has 223..447) 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 3 client.124136.objecter handle_osd_map got epochs [447,447] > 0 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 3 client.124136.objecter handle_osd_map decoding full epoch 447 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 20 client.124136.objecter dump_active .. 0 homeless 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3682695872 0 0) 0x7ff1b40065d0 con 0x7ff1b4002870 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient(hunting): my global_id is 114237 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1d409d180 con 0x7ff1b4002870 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4022118774 0 0) 0x7ff1d409d180 con 0x7ff1b4002870 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1b40065d0 con 0x7ff1b4002870 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1c4004da0 con 0x7ff1b4002870 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.707+0000 7ff1c9ffb640 10 monclient: dump: 2024-09-15T20:43:04.732 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1d9d9d640 10 monclient: get_auth_request con 0x7ff1b803dfd0 auth_method 0 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2724255396 0 0) 0x7ff1b40065d0 con 0x7ff1b4002870 2024-09-15T20:43:04.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff1b400a170 con 0x7ff1b4002870 2024-09-15T20:43:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1c9ffb640 10 client.124136.objecter ms_handle_connect 0x7ff1b803dfd0 2024-09-15T20:43:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff1c4007480 con 0x7ff1b4002870 2024-09-15T20:43:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1c9ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1c9ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 >> v1:172.21.15.78:6789/0 conn(0x7ff1b4002870 legacy=0x7ff1b4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.711+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 >> v1:172.21.15.78:6789/0 conn(0x7ff1b4002870 legacy=0x7ff1b4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:43:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff1b4009e60 con 0x7ff1b4009120 2024-09-15T20:43:04.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 client.124136.objecter ms_handle_connect 0x7ff1b4009120 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 client.124136.objecter resend_mon_ops 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1573807729 0 0) 0x7ff1b4009e60 con 0x7ff1b4009120 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient(hunting): my global_id is 114240 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff1b40065d0 con 0x7ff1b4009120 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1310958873 0 0) 0x7ff1b40065d0 con 0x7ff1b4009120 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff1b4009e60 con 0x7ff1b4009120 2024-09-15T20:43:04.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff1c4005490 con 0x7ff1b4009120 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient: dump: 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.737 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2864519024 0 0) 0x7ff1b4009e60 con 0x7ff1b4009120 2024-09-15T20:43:04.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1b400c400 con 0x7ff1b4009120 2024-09-15T20:43:04.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4140 (unknown 0 0 3300158974) 0x7ff1c40076b0 con 0x7ff1b4009120 2024-09-15T20:43:04.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:43:04.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:04.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 >> v1:172.21.15.78:6789/0 conn(0x7ff1b4009120 legacy=0x7ff1b40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1c9ffb640 1 -- 172.21.15.78:0/3900816455 >> v1:172.21.15.78:6789/0 conn(0x7ff1b4009120 legacy=0x7ff1b40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:04.741 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:43:04.741 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 44, 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 110558, 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:04.742 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:04.743 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:04.744 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:43:04.745 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:04.746 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:04.747 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.748 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:04.749 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:43:04.750 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.751 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.752 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.753 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:43:04.754 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:43:04.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 10 librados: watch_flush enter 2024-09-15T20:43:04.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 10 librados: watch_flush exit 2024-09-15T20:43:04.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 20 client.124136.objecter shutdown clearing up homeless session... 2024-09-15T20:43:04.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 10 client.124136.objecter successfully canceled tick 2024-09-15T20:43:04.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 >> v1:172.21.15.146:6812/435598642 conn(0x7ff1b803dfd0 legacy=0x7ff1b8040490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 10 monclient: shutdown 2024-09-15T20:43:04.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 >> v1:172.21.15.78:6790/0 conn(0x7ff1d4135b60 legacy=0x7ff1d4072460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.715+0000 7ff1d9d9d640 1 -- 172.21.15.78:0/3900816455 reap_dead start 2024-09-15T20:43:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.719+0000 7ff1d9d9d640 1 -- 172.21.15.78:0/3900816455 reap_dead start 2024-09-15T20:43:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.719+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 shutdown_connections 2024-09-15T20:43:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.719+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 >> 172.21.15.78:0/3900816455 conn(0x7ff1d40a63c0 msgr2=0x7ff1d40a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.719+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 shutdown_connections 2024-09-15T20:43:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.719+0000 7ff1db800640 1 -- 172.21.15.78:0/3900816455 wait complete. 2024-09-15T20:43:04.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.719+0000 7ff1db800640 1 librados: shutdown 2024-09-15T20:43:04.757 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:43:04.757 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:43:04.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:43:04.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient: get_monmap_and_config 2024-09-15T20:43:04.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient: build_initial_monmap 2024-09-15T20:43:04.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient: monmap: 2024-09-15T20:43:04.869 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:04.869 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.869 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:04.869938+0000 2024-09-15T20:43:04.869 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:04.869938+0000 2024-09-15T20:43:04.869 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:04.870 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.870 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:04.870 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:04.870 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:04.870 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 1 Processor -- start 2024-09-15T20:43:04.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 1 -- start start 2024-09-15T20:43:04.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient: init 2024-09-15T20:43:04.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:04.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient(hunting): picked mon.noname-b con 0x7fdd700a0520 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient(hunting): picked mon.noname-a con 0x7fdd701347a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient(hunting): picked mon.noname-c con 0x7fdd7013e280 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd700738b0 con 0x7fdd701347a0 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd700744d0 con 0x7fdd7013e280 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd700747b0 con 0x7fdd700a0520 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 10 monclient(hunting): authenticate will time out at 1845.755158s 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd65d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdd701347a0 0x7fdd7013cb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55630/0 (socket says 172.21.15.78:55630) 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd65d9b640 1 -- 172.21.15.78:0/1152572310 learned_addr learned my addr 172.21.15.78:0/1152572310 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6e59c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fdd700a0520 0x7fdd700a0900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50932/0 (socket says 172.21.15.78:50932) 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 280447244 0 0) 0x7fdd700738b0 con 0x7fdd701347a0 2024-09-15T20:43:04.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient(hunting): my global_id is 114243 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd5c003530 con 0x7fdd701347a0 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2204759537 0 0) 0x7fdd700747b0 con 0x7fdd700a0520 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient(hunting): my global_id is 124139 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd700738b0 con 0x7fdd700a0520 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2204466844 0 0) 0x7fdd5c003530 con 0x7fdd701347a0 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 >> v1:172.21.15.78:6790/0 conn(0x7fdd700a0520 legacy=0x7fdd700a0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 >> v1:172.21.15.146:6789/0 conn(0x7fdd7013e280 legacy=0x7fdd70140670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient: _finish_hunting 0 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 monclient: found mon.noname-a 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7009cac0 con 0x7fdd701347a0 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient: _finish_auth 0 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient: _check_auth_tickets 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd580016a0 con 0x7fdd701347a0 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd6dd9b640 10 monclient: dump: 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.867+0000 7fdd74b48640 5 monclient: authenticate success, global_id 114243 2024-09-15T20:43:04.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd580018c0 con 0x7fdd701347a0 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd6dd9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd6dd9b640 1 -- 172.21.15.78:0/1152572310 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd58001c90 con 0x7fdd701347a0 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 4 set_mon_vals no callback set 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd6dd9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd6dd9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd6dd9b640 10 monclient: dump: 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.885 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: shutdown 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- 172.21.15.78:0/1152572310 >> v1:172.21.15.78:6789/0 conn(0x7fdd701347a0 legacy=0x7fdd7013cb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- 172.21.15.78:0/1152572310 shutdown_connections 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- 172.21.15.78:0/1152572310 >> 172.21.15.78:0/1152572310 conn(0x7fdd700a3380 msgr2=0x7fdd700a3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:04.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- 172.21.15.78:0/1152572310 shutdown_connections 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- 172.21.15.78:0/1152572310 wait complete. 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: build_initial_monmap 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: monmap: 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:04.887 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.888 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:04.888 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:04.888 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:04.888 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 librados: starting msgr at 2024-09-15T20:43:04.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 librados: starting objecter 2024-09-15T20:43:04.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 Processor -- start 2024-09-15T20:43:04.888 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- start start 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 librados: setting wanted keys 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 librados: calling monclient init 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: init 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient(hunting): picked mon.noname-b con 0x7fdd700a0520 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient(hunting): picked mon.noname-a con 0x7fdd701347a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient(hunting): picked mon.noname-c con 0x7fdd7013e280 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:04.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd70139aa0 con 0x7fdd701347a0 2024-09-15T20:43:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd70139d80 con 0x7fdd700a0520 2024-09-15T20:43:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd70138580 con 0x7fdd7013e280 2024-09-15T20:43:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient(hunting): authenticate will time out at 1845.758214s 2024-09-15T20:43:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd65d9b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdd701347a0 0x7fdd701d1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55638/0 (socket says 172.21.15.78:55638) 2024-09-15T20:43:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd65d9b640 1 -- 172.21.15.78:0/727922175 learned_addr learned my addr 172.21.15.78:0/727922175 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd6ed9d640 1 --1- 172.21.15.78:0/727922175 >> v1:172.21.15.78:6790/0 conn(0x7fdd7013e280 0x7fdd701d5670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50948/0 (socket says 172.21.15.78:50948) 2024-09-15T20:43:04.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 client.?.objecter ms_handle_connect 0x7fdd701347a0 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 client.?.objecter ms_handle_connect 0x7fdd7013e280 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2467926042 0 0) 0x7fdd70138580 con 0x7fdd7013e280 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): my global_id is 124142 2024-09-15T20:43:04.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd50004ef0 con 0x7fdd7013e280 2024-09-15T20:43:04.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 115019505 0 0) 0x7fdd70139aa0 con 0x7fdd701347a0 2024-09-15T20:43:04.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): my global_id is 114246 2024-09-15T20:43:04.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd70138580 con 0x7fdd701347a0 2024-09-15T20:43:04.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 client.?.objecter ms_handle_connect 0x7fdd700a0520 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1394454893 0 0) 0x7fdd50004ef0 con 0x7fdd7013e280 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd70139aa0 con 0x7fdd7013e280 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd68001c80 con 0x7fdd7013e280 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): dump: 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.893 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.894 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.894 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.894 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.894 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.894 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.894 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.894 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.894 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2388085847 0 0) 0x7fdd70138580 con 0x7fdd701347a0 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd50004ef0 con 0x7fdd701347a0 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd58001c10 con 0x7fdd701347a0 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient(hunting): dump: 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.895 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1953338661 0 0) 0x7fdd70139aa0 con 0x7fdd7013e280 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 >> v1:172.21.15.146:6789/0 conn(0x7fdd700a0520 legacy=0x7fdd701ccf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 >> v1:172.21.15.78:6789/0 conn(0x7fdd701347a0 legacy=0x7fdd701d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient: _finish_hunting 0 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 monclient: found mon.c 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd7013a7b0 con 0x7fdd7013e280 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient: _finish_auth 0 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient: _check_auth_tickets 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 5 monclient: authenticate success, global_id 124142 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: _renew_subs 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdd70139220 con 0x7fdd7013e280 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 client.124142.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:04.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: _renew_subs 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdd70136fb0 con 0x7fdd7013e280 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 librados: init done 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdd680039a0 con 0x7fdd7013e280 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd68003db0 con 0x7fdd7013e280 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd6d59a640 4 set_mon_vals no callback set 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd677fe640 10 monclient: dump: 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.898 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.871+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdd540065d0 con 0x7fdd54002870 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fdd68003f50 con 0x7fdd7013e280 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 client.124142.objecter ms_handle_connect 0x7fdd54002870 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 client.124142.objecter resend_mon_ops 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(447..447 src has 223..447) ==== 5354+0+0 (unknown 3060371923 0 0) 0x7fdd6804b9e0 con 0x7fdd7013e280 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 client.124142.objecter ms_dispatch 0x7fdd70000ba0 osd_map(447..447 src has 223..447) 2024-09-15T20:43:04.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 3 client.124142.objecter handle_osd_map got epochs [447,447] > 0 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 3 client.124142.objecter handle_osd_map decoding full epoch 447 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 20 client.124142.objecter dump_active .. 0 homeless 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2188258140 0 0) 0x7fdd540065d0 con 0x7fdd54002870 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient(hunting): my global_id is 124145 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd70139aa0 con 0x7fdd54002870 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd65d9b640 10 monclient: get_auth_request con 0x7fdd5003e070 auth_method 0 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4049505952 0 0) 0x7fdd70139aa0 con 0x7fdd54002870 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd540065d0 con 0x7fdd54002870 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd60001750 con 0x7fdd54002870 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient: dump: 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4245123607 0 0) 0x7fdd540065d0 con 0x7fdd54002870 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd5400a170 con 0x7fdd54002870 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 client.124142.objecter ms_handle_connect 0x7fdd5003e070 2024-09-15T20:43:04.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdd60001970 con 0x7fdd54002870 2024-09-15T20:43:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 >> v1:172.21.15.78:6790/0 conn(0x7fdd54002870 legacy=0x7fdd54004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.875+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 >> v1:172.21.15.78:6790/0 conn(0x7fdd54002870 legacy=0x7fdd54004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd74b48640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:43:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdd54009e80 con 0x7fdd54009140 2024-09-15T20:43:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 client.124142.objecter ms_handle_connect 0x7fdd54009140 2024-09-15T20:43:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 client.124142.objecter resend_mon_ops 2024-09-15T20:43:04.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2700444380 0 0) 0x7fdd54009e80 con 0x7fdd54009140 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 monclient(hunting): my global_id is 124148 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdd540065d0 con 0x7fdd54009140 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3201347465 0 0) 0x7fdd540065d0 con 0x7fdd54009140 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdd54009e80 con 0x7fdd54009140 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdd60004990 con 0x7fdd54009140 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.879+0000 7fdd677fe640 10 monclient: dump: 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:04.904 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3718304632 0 0) 0x7fdd54009e80 con 0x7fdd54009140 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdd5400c420 con 0x7fdd54009140 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3969 (unknown 0 0 2998994062) 0x7fdd600047f0 con 0x7fdd54009140 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd677fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd677fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 >> v1:172.21.15.78:6790/0 conn(0x7fdd54009140 legacy=0x7fdd540095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd677fe640 1 -- 172.21.15.78:0/727922175 >> v1:172.21.15.78:6790/0 conn(0x7fdd54009140 legacy=0x7fdd540095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 44, 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 2811, 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:43:04.906 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:04.907 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:04.908 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:04.909 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:04.910 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:43:04.911 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:43:04.912 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:04.913 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:04.914 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:04.916 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:43:04.917 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:04.917 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:04.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:04.917 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:43:04.917 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:43:04.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 10 librados: watch_flush enter 2024-09-15T20:43:04.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 10 librados: watch_flush exit 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 20 client.124142.objecter shutdown clearing up homeless session... 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 10 client.124142.objecter successfully canceled tick 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 >> v1:172.21.15.146:6812/435598642 conn(0x7fdd5003e070 legacy=0x7fdd50040530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 10 monclient: shutdown 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 >> v1:172.21.15.78:6790/0 conn(0x7fdd7013e280 legacy=0x7fdd701d5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd6ed9d640 1 -- 172.21.15.78:0/727922175 reap_dead start 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd6ed9d640 1 -- 172.21.15.78:0/727922175 reap_dead start 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 shutdown_connections 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 >> 172.21.15.78:0/727922175 conn(0x7fdd700a3380 msgr2=0x7fdd700a6120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 shutdown_connections 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 1 -- 172.21.15.78:0/727922175 wait complete. 2024-09-15T20:43:04.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:04.883+0000 7fdd74b48640 1 librados: shutdown 2024-09-15T20:43:04.920 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:43:04.920 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:43:04.920 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:43:05.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient: get_monmap_and_config 2024-09-15T20:43:05.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient: build_initial_monmap 2024-09-15T20:43:05.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient: monmap: 2024-09-15T20:43:05.045 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:05.045 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.045 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:05.037650+0000 2024-09-15T20:43:05.045 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:05.037650+0000 2024-09-15T20:43:05.045 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:05.045 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 Processor -- start 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- start start 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient: init 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient(hunting): picked mon.noname-a con 0x7f8d080a0ff0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient(hunting): picked mon.noname-b con 0x7f8d08073800 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient(hunting): picked mon.noname-c con 0x7f8d080a1ae0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d08072df0 con 0x7f8d080a0ff0 2024-09-15T20:43:05.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d08074110 con 0x7f8d080a1ae0 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d080743f0 con 0x7f8d08073800 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient(hunting): authenticate will time out at 1845.922193s 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0da03640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8d080a0ff0 0x7f8d080a13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55654/0 (socket says 172.21.15.78:55654) 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0da03640 1 -- 172.21.15.78:0/1942189884 learned_addr learned my addr 172.21.15.78:0/1942189884 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2848311419 0 0) 0x7f8d08072df0 con 0x7f8d080a0ff0 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient(hunting): my global_id is 114249 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cf4003530 con 0x7f8d080a0ff0 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1223129493 0 0) 0x7f8d080743f0 con 0x7f8d08073800 2024-09-15T20:43:05.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient(hunting): my global_id is 124151 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d08072df0 con 0x7f8d08073800 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1459766712 0 0) 0x7f8cf4003530 con 0x7f8d080a0ff0 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 >> v1:172.21.15.78:6790/0 conn(0x7f8d08073800 legacy=0x7f8d0813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 >> v1:172.21.15.146:6789/0 conn(0x7f8d080a1ae0 legacy=0x7f8d08140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: _finish_hunting 0 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 monclient: found mon.noname-a 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d08074eb0 con 0x7f8d080a0ff0 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: _finish_auth 0 2024-09-15T20:43:05.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: _check_auth_tickets 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8d040016a0 con 0x7f8d080a0ff0 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: dump: 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.049 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 5 monclient: authenticate success, global_id 114249 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d04001840 con 0x7f8d080a0ff0 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 1 -- 172.21.15.78:0/1942189884 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8d04001c50 con 0x7f8d080a0ff0 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 4 set_mon_vals no callback set 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0ca01640 10 monclient: dump: 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.050 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.051 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.051 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.051 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.051 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.052 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.052 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.052 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 10 monclient: shutdown 2024-09-15T20:43:05.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:05.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- 172.21.15.78:0/1942189884 >> v1:172.21.15.78:6789/0 conn(0x7f8d080a0ff0 legacy=0x7f8d080a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- 172.21.15.78:0/1942189884 shutdown_connections 2024-09-15T20:43:05.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- 172.21.15.78:0/1942189884 >> 172.21.15.78:0/1942189884 conn(0x7f8d080a63c0 msgr2=0x7f8d080a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:05.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- 172.21.15.78:0/1942189884 shutdown_connections 2024-09-15T20:43:05.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.035+0000 7f8d0fc67640 1 -- 172.21.15.78:0/1942189884 wait complete. 2024-09-15T20:43:05.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: build_initial_monmap 2024-09-15T20:43:05.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: monmap: 2024-09-15T20:43:05.054 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:05.054 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:05.054 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:05.054 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:05.054 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:05.054 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.054 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 librados: starting msgr at 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 librados: starting objecter 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 Processor -- start 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 -- start start 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 librados: setting wanted keys 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 librados: calling monclient init 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: init 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:05.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient(hunting): picked mon.noname-a con 0x7f8d08073800 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient(hunting): picked mon.noname-b con 0x7f8d080a0ff0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient(hunting): picked mon.noname-c con 0x7f8d080a1ae0 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0813a690 con 0x7f8d08073800 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d081375a0 con 0x7f8d080a0ff0 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0da03640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8d08073800 0x7f8d08136780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55662/0 (socket says 172.21.15.78:55662) 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0da03640 1 -- 172.21.15.78:0/3762596798 learned_addr learned my addr 172.21.15.78:0/3762596798 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d0809d180 con 0x7f8d080a1ae0 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient(hunting): authenticate will time out at 1845.925647s 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.?.objecter ms_handle_connect 0x7f8d08073800 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:05.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.?.objecter ms_handle_connect 0x7f8d080a1ae0 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3467671245 0 0) 0x7f8d0809d180 con 0x7f8d080a1ae0 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): my global_id is 124154 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cec004ec0 con 0x7f8d080a1ae0 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3253713373 0 0) 0x7f8d08074eb0 con 0x7f8d08073800 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): my global_id is 114252 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d0809d180 con 0x7f8d08073800 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.?.objecter ms_handle_connect 0x7f8d080a0ff0 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1112947764 0 0) 0x7f8cec004ec0 con 0x7f8d080a1ae0 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8d08074eb0 con 0x7f8d080a1ae0 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8d00001c40 con 0x7f8d080a1ae0 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): dump: 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.058 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2806955406 0 0) 0x7f8d0809d180 con 0x7f8d08073800 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8cec004ec0 con 0x7f8d08073800 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8d04001b90 con 0x7f8d08073800 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:05.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): dump: 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1897007176 0 0) 0x7f8d081375a0 con 0x7f8d080a0ff0 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient(hunting): my global_id is 23728 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d0809d180 con 0x7f8d080a0ff0 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3995948451 0 0) 0x7f8d08074eb0 con 0x7f8d080a1ae0 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 >> v1:172.21.15.146:6789/0 conn(0x7f8d080a0ff0 legacy=0x7f8d08136e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 >> v1:172.21.15.78:6789/0 conn(0x7f8d08073800 legacy=0x7f8d08136780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 monclient: found mon.c 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d0809db30 con 0x7f8d080a1ae0 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient: _finish_auth 0 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:43:05.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:05.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 5 monclient: authenticate success, global_id 124154 2024-09-15T20:43:05.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: _renew_subs 2024-09-15T20:43:05.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:05.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8d0813af30 con 0x7f8d080a1ae0 2024-09-15T20:43:05.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 client.124154.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:05.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: _renew_subs 2024-09-15T20:43:05.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:05.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8d08139990 con 0x7f8d080a1ae0 2024-09-15T20:43:05.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 librados: init done 2024-09-15T20:43:05.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8d00003980 con 0x7f8d080a1ae0 2024-09-15T20:43:05.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:05.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8d00003cd0 con 0x7f8d080a1ae0 2024-09-15T20:43:05.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:05.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:05.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfffff640 4 set_mon_vals no callback set 2024-09-15T20:43:05.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 monclient: dump: 2024-09-15T20:43:05.064 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.065 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8cf00065d0 con 0x7f8cf0002870 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f8d0000fd40 con 0x7f8d080a1ae0 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.124154.objecter ms_handle_connect 0x7f8cf0002870 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.124154.objecter resend_mon_ops 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(447..447 src has 223..447) ==== 5354+0+0 (unknown 3060371923 0 0) 0x7f8d0004a880 con 0x7f8d080a1ae0 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 10 client.124154.objecter ms_dispatch 0x7f8d08000ba0 osd_map(447..447 src has 223..447) 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 3 client.124154.objecter handle_osd_map got epochs [447,447] > 0 2024-09-15T20:43:05.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.039+0000 7f8cfe7fc640 3 client.124154.objecter handle_osd_map decoding full epoch 447 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 20 client.124154.objecter dump_active .. 0 homeless 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2732084740 0 0) 0x7f8cf00065d0 con 0x7f8cf0002870 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient(hunting): my global_id is 23731 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d08074eb0 con 0x7f8cf0002870 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8d0d202640 10 monclient: get_auth_request con 0x7f8cec03e270 auth_method 0 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 client.124154.objecter ms_handle_connect 0x7f8cec03e270 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2379737119 0 0) 0x7f8d08074eb0 con 0x7f8cf0002870 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8cf00065d0 con 0x7f8cf0002870 2024-09-15T20:43:05.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8d04001820 con 0x7f8cf0002870 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient: dump: 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4077414150 0 0) 0x7f8cf00065d0 con 0x7f8cf0002870 2024-09-15T20:43:05.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8cf000a170 con 0x7f8cf0002870 2024-09-15T20:43:05.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f8d04006be0 con 0x7f8cf0002870 2024-09-15T20:43:05.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:05.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:05.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 >> v1:172.21.15.146:6789/0 conn(0x7f8cf0002870 legacy=0x7f8cf0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.043+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 >> v1:172.21.15.146:6789/0 conn(0x7f8cf0002870 legacy=0x7f8cf0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:05.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.047+0000 7f8d0fc67640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:43:05.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.047+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f8cf0009e30 con 0x7f8cf00090f0 2024-09-15T20:43:05.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 client.124154.objecter ms_handle_connect 0x7f8cf00090f0 2024-09-15T20:43:05.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 client.124154.objecter resend_mon_ops 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3574143976 0 0) 0x7f8cf0009e30 con 0x7f8cf00090f0 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient(hunting): my global_id is 23734 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8cf00065d0 con 0x7f8cf00090f0 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 93367737 0 0) 0x7f8cf00065d0 con 0x7f8cf00090f0 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8cf0009e30 con 0x7f8cf00090f0 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8d04004f90 con 0x7f8cf00090f0 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:43:05.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient: dump: 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1438191241 0 0) 0x7f8cf0009e30 con 0x7f8cf00090f0 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8cf000c400 con 0x7f8cf00090f0 2024-09-15T20:43:05.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 2755000649) 0x7f8d04006420 con 0x7f8cf00090f0 2024-09-15T20:43:05.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:43:05.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:05.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 >> v1:172.21.15.146:6789/0 conn(0x7f8cf00090f0 legacy=0x7f8cf0009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8cfe7fc640 1 -- 172.21.15.78:0/3762596798 >> v1:172.21.15.146:6789/0 conn(0x7f8cf00090f0 legacy=0x7f8cf0009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:05.073 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:43:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:43:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 44, 2024-09-15T20:43:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 697560, 2024-09-15T20:43:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:43:05.074 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:05.075 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:43:05.076 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:05.077 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:05.078 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:05.079 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:05.080 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:05.081 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:43:05.082 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:43:05.083 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:43:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:43:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:05.084 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:43:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:43:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.085 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:05.086 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:05.087 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:05.088 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:43:05.088 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:43:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 10 librados: watch_flush enter 2024-09-15T20:43:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 10 librados: watch_flush exit 2024-09-15T20:43:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 20 client.124154.objecter shutdown clearing up homeless session... 2024-09-15T20:43:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 10 client.124154.objecter successfully canceled tick 2024-09-15T20:43:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 >> v1:172.21.15.146:6812/435598642 conn(0x7f8cec03e270 legacy=0x7f8cec040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 10 monclient: shutdown 2024-09-15T20:43:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:05.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 >> v1:172.21.15.78:6790/0 conn(0x7f8d080a1ae0 legacy=0x7f8d081d4f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0e204640 1 -- 172.21.15.78:0/3762596798 reap_dead start 2024-09-15T20:43:05.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0e204640 1 -- 172.21.15.78:0/3762596798 reap_dead start 2024-09-15T20:43:05.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 shutdown_connections 2024-09-15T20:43:05.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 >> 172.21.15.78:0/3762596798 conn(0x7f8d080a63c0 msgr2=0x7f8d08138220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:05.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 shutdown_connections 2024-09-15T20:43:05.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 1 -- 172.21.15.78:0/3762596798 wait complete. 2024-09-15T20:43:05.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.051+0000 7f8d0fc67640 1 librados: shutdown 2024-09-15T20:43:05.093 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:43:05.093 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T20:43:05.093 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T20:43:05.093 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T20:43:05.093 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T20:43:05.093 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T20:43:05.094 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:43:05.126 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T20:43:05.126 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:43:05.126 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:43:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: get_monmap_and_config 2024-09-15T20:43:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: build_initial_monmap 2024-09-15T20:43:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: monmap: 2024-09-15T20:43:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:05.247 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:05.241021+0000 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:05.241021+0000 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 Processor -- start 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- start start 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: init 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:43:05.248 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient(hunting): picked mon.noname-b con 0x7fc454137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient(hunting): picked mon.noname-c con 0x7fc4540a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient(hunting): picked mon.noname-a con 0x7fc45413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4540a6420 con 0x7fc45413e5f0 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4540a6700 con 0x7fc4540a9610 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4540a2110 con 0x7fc454137ce0 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient(hunting): authenticate will time out at 1846.125604s 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc452ffd640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fc4540a9610 legacy=0x7fc45413ced0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4537fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc454137ce0 0x7fc4541380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51028/0 (socket says 172.21.15.78:51028) 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4537fe640 1 -- 172.21.15.78:0/1277916329 learned_addr learned my addr 172.21.15.78:0/1277916329 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc453fff640 1 --1- 172.21.15.78:0/1277916329 >> v1:172.21.15.78:6789/0 conn(0x7fc45413e5f0 0x7fc4541409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55680/0 (socket says 172.21.15.78:55680) 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3298308526 0 0) 0x7fc4540a6420 con 0x7fc45413e5f0 2024-09-15T20:43:05.249 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient(hunting): my global_id is 114255 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc43c003530 con 0x7fc45413e5f0 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2663254098 0 0) 0x7fc43c003530 con 0x7fc45413e5f0 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 >> v1:172.21.15.78:6790/0 conn(0x7fc454137ce0 legacy=0x7fc4541380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 >> v1:172.21.15.146:6789/0 conn(0x7fc4540a9610 legacy=0x7fc45413ced0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: _finish_hunting 0 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 monclient: found mon.noname-a 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4540a4bc0 con 0x7fc45413e5f0 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: _finish_auth 0 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: _check_auth_tickets 2024-09-15T20:43:05.250 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc44c001950 con 0x7fc45413e5f0 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 5 monclient: authenticate success, global_id 114255 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: dump: 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.251 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc44c001b70 con 0x7fc45413e5f0 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 1 -- 172.21.15.78:0/1277916329 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc44c001f80 con 0x7fc45413e5f0 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 4 set_mon_vals no callback set 2024-09-15T20:43:05.252 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc4527fc640 10 monclient: dump: 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.253 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: shutdown 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- 172.21.15.78:0/1277916329 >> v1:172.21.15.78:6789/0 conn(0x7fc45413e5f0 legacy=0x7fc4541409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- 172.21.15.78:0/1277916329 shutdown_connections 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- 172.21.15.78:0/1277916329 >> 172.21.15.78:0/1277916329 conn(0x7fc4540aa6c0 msgr2=0x7fc4540aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- 172.21.15.78:0/1277916329 shutdown_connections 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 1 -- 172.21.15.78:0/1277916329 wait complete. 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: build_initial_monmap 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.239+0000 7fc459f13640 10 monclient: monmap: 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:05.254 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 librados: starting msgr at 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 librados: starting objecter 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 Processor -- start 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 -- start start 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 librados: setting wanted keys 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 librados: calling monclient init 2024-09-15T20:43:05.255 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: init 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient(hunting): picked mon.noname-c con 0x7fc4540a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient(hunting): picked mon.noname-b con 0x7fc454137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient(hunting): picked mon.noname-a con 0x7fc45413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc45413bfd0 con 0x7fc45413e5f0 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc454139cd0 con 0x7fc454137ce0 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc45413c2b0 con 0x7fc4540a9610 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient(hunting): authenticate will time out at 1846.128715s 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc4537fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc4540a9610 0x7fc454138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51036/0 (socket says 172.21.15.78:51036) 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc4537fe640 1 -- 172.21.15.78:0/4152942557 learned_addr learned my addr 172.21.15.78:0/4152942557 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc453fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc45413e5f0 0x7fc4541d4f50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55682/0 (socket says 172.21.15.78:55682) 2024-09-15T20:43:05.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc452ffd640 1 -- 172.21.15.78:0/4152942557 >> v1:172.21.15.146:6789/0 conn(0x7fc454137ce0 legacy=0x7fc4541395c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 client.?.objecter ms_handle_connect 0x7fc4540a9610 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 client.?.objecter ms_handle_connect 0x7fc45413e5f0 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2540309945 0 0) 0x7fc45413bfd0 con 0x7fc45413e5f0 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient(hunting): my global_id is 114258 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc440004ec0 con 0x7fc45413e5f0 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2638594153 0 0) 0x7fc440004ec0 con 0x7fc45413e5f0 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc45413bfd0 con 0x7fc45413e5f0 2024-09-15T20:43:05.257 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc44c001ec0 con 0x7fc45413e5f0 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient(hunting): dump: 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.258 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3603134843 0 0) 0x7fc45413bfd0 con 0x7fc45413e5f0 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 >> v1:172.21.15.78:6790/0 conn(0x7fc4540a9610 legacy=0x7fc454138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 >> v1:172.21.15.146:6789/0 conn(0x7fc454137ce0 legacy=0x7fc4541395c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 monclient: found mon.a 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4540743e0 con 0x7fc45413e5f0 2024-09-15T20:43:05.259 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient: _finish_auth 0 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 5 monclient: authenticate success, global_id 114258 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: _renew_subs 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc4540a4030 con 0x7fc45413e5f0 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 client.114258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: _renew_subs 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc45413ae60 con 0x7fc45413e5f0 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 librados: init done 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc44c001ad0 con 0x7fc45413e5f0 2024-09-15T20:43:05.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc44c004fc0 con 0x7fc45413e5f0 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc451ffb640 4 set_mon_vals no callback set 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 monclient: dump: 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:05.261 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:05.262 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:05.262 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:05.262 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:05.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:05.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:05.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:05.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc434005090 con 0x7fc45413e5f0 2024-09-15T20:43:05.262 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fc44c011030 con 0x7fc45413e5f0 2024-09-15T20:43:05.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(447..447 src has 223..447) ==== 5354+0+0 (unknown 3060371923 0 0) 0x7fc44c04bae0 con 0x7fc45413e5f0 2024-09-15T20:43:05.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 client.114258.objecter ms_dispatch 0x7fc454000ba0 osd_map(447..447 src has 223..447) 2024-09-15T20:43:05.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 3 client.114258.objecter handle_osd_map got epochs [447,447] > 0 2024-09-15T20:43:05.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 3 client.114258.objecter handle_osd_map decoding full epoch 447 2024-09-15T20:43:05.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 20 client.114258.objecter dump_active .. 0 homeless 2024-09-15T20:43:05.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc4537fe640 10 monclient: get_auth_request con 0x7fc44003df20 auth_method 0 2024-09-15T20:43:05.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.243+0000 7fc43bfff640 10 client.114258.objecter ms_handle_connect 0x7fc44003df20 2024-09-15T20:43:05.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.247+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc44c016e40 con 0x7fc45413e5f0 2024-09-15T20:43:05.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.247+0000 7fc43bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:05.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.247+0000 7fc43bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:05.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:43:05.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:05.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:05.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc434005380 con 0x7fc45413e5f0 2024-09-15T20:43:05.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc43bfff640 1 -- 172.21.15.78:0/4152942557 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3815910849) 0x7fc44c016740 con 0x7fc45413e5f0 2024-09-15T20:43:05.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc43bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:05.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc43bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:05.485 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:43:05.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 10 librados: watch_flush enter 2024-09-15T20:43:05.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 10 librados: watch_flush exit 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 20 client.114258.objecter shutdown clearing up homeless session... 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 10 client.114258.objecter successfully canceled tick 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 >> v1:172.21.15.146:6812/435598642 conn(0x7fc44003df20 legacy=0x7fc4400403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 10 monclient: shutdown 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 >> v1:172.21.15.78:6789/0 conn(0x7fc45413e5f0 legacy=0x7fc4541d4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 shutdown_connections 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 >> 172.21.15.78:0/4152942557 conn(0x7fc4540aa6c0 msgr2=0x7fc4541409e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 shutdown_connections 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.483+0000 7fc459f13640 1 -- 172.21.15.78:0/4152942557 wait complete. 2024-09-15T20:43:05.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:05.487+0000 7fc459f13640 1 librados: shutdown 2024-09-15T20:43:08.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:43:08.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient: get_monmap_and_config 2024-09-15T20:43:08.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient: build_initial_monmap 2024-09-15T20:43:08.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient: monmap: 2024-09-15T20:43:08.629 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:08.629 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:08.629 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:08.630445+0000 2024-09-15T20:43:08.629 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:08.630445+0000 2024-09-15T20:43:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:08.630 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:08.630 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 1 Processor -- start 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 1 -- start start 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient: init 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient(hunting): picked mon.noname-b con 0x7f30f4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient(hunting): picked mon.noname-c con 0x7f30f40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient(hunting): picked mon.noname-a con 0x7f30f413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30f40745e0 con 0x7f30f413c220 2024-09-15T20:43:08.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30f406ecf0 con 0x7f30f40a24e0 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30f406efd0 con 0x7f30f4135b60 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f9fb6640 10 monclient(hunting): authenticate will time out at 1849.515276s 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f37fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f30f4135b60 0x7f30f4135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51050/0 (socket says 172.21.15.78:51050) 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f37fe640 1 -- 172.21.15.78:0/4189060471 learned_addr learned my addr 172.21.15.78:0/4189060471 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f2ffd640 1 -- 172.21.15.78:0/4189060471 >> v1:172.21.15.146:6789/0 conn(0x7f30f40a24e0 legacy=0x7f30f413ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1069517513 0 0) 0x7f30f40745e0 con 0x7f30f413c220 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:08.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient(hunting): my global_id is 114261 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30e0003530 con 0x7f30f413c220 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3819506665 0 0) 0x7f30e0003530 con 0x7f30f413c220 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 >> v1:172.21.15.78:6790/0 conn(0x7f30f4135b60 legacy=0x7f30f4135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 >> v1:172.21.15.146:6789/0 conn(0x7f30f40a24e0 legacy=0x7f30f413ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient: _finish_hunting 0 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 1 monclient: found mon.noname-a 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f4136650 con 0x7f30f413c220 2024-09-15T20:43:08.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient: _finish_auth 0 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient: _check_auth_tickets 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f30ec001a20 con 0x7f30f413c220 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.627+0000 7f30f27fc640 10 monclient: dump: 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:08.640 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 5 monclient: authenticate success, global_id 114261 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30ec001c40 con 0x7f30f413c220 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f27fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f27fc640 1 -- 172.21.15.78:0/4189060471 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f30ec002050 con 0x7f30f413c220 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:08.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f27fc640 10 monclient: dump: 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:08.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 4 set_mon_vals no callback set 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: shutdown 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- 172.21.15.78:0/4189060471 >> v1:172.21.15.78:6789/0 conn(0x7f30f413c220 legacy=0x7f30f413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- 172.21.15.78:0/4189060471 shutdown_connections 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- 172.21.15.78:0/4189060471 >> 172.21.15.78:0/4189060471 conn(0x7f30f40a63c0 msgr2=0x7f30f40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- 172.21.15.78:0/4189060471 shutdown_connections 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- 172.21.15.78:0/4189060471 wait complete. 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: build_initial_monmap 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: monmap: 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:08.643 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 librados: starting msgr at 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 librados: starting objecter 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 Processor -- start 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- start start 2024-09-15T20:43:08.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 librados: setting wanted keys 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 librados: calling monclient init 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: init 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient(hunting): picked mon.noname-b con 0x7f30f40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient(hunting): picked mon.noname-c con 0x7f30f4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient(hunting): picked mon.noname-a con 0x7f30f413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30f413f900 con 0x7f30f413c220 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30f4072ae0 con 0x7f30f40a24e0 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30f413b770 con 0x7f30f4135b60 2024-09-15T20:43:08.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient(hunting): authenticate will time out at 1849.518692s 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f30f4135b60 0x7f30f409dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:51058/0 (socket says 172.21.15.78:51058) 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f37fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7f30f40a24e0 legacy=0x7f30f409d490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f2ffd640 1 -- 172.21.15.78:0/1851155297 learned_addr learned my addr 172.21.15.78:0/1851155297 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f3fff640 1 --1- 172.21.15.78:0/1851155297 >> v1:172.21.15.78:6789/0 conn(0x7f30f413c220 0x7f30f41d4c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:55706/0 (socket says 172.21.15.78:55706) 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 client.?.objecter ms_handle_connect 0x7f30f413c220 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 client.?.objecter ms_handle_connect 0x7f30f4135b60 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 838024926 0 0) 0x7f30f4136650 con 0x7f30f413c220 2024-09-15T20:43:08.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient(hunting): my global_id is 114264 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f30d4004ec0 con 0x7f30f413c220 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2129432697 0 0) 0x7f30d4004ec0 con 0x7f30f413c220 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f30f4136650 con 0x7f30f413c220 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f30ec001f90 con 0x7f30f413c220 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient(hunting): dump: 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:08.647 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2559204918 0 0) 0x7f30f4136650 con 0x7f30f413c220 2024-09-15T20:43:08.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 >> v1:172.21.15.78:6790/0 conn(0x7f30f4135b60 legacy=0x7f30f409dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 >> v1:172.21.15.146:6789/0 conn(0x7f30f40a24e0 legacy=0x7f30f409d490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient: _finish_hunting 0 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 monclient: found mon.a 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30f409e2b0 con 0x7f30f413c220 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient: _finish_auth 0 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient: _check_auth_tickets 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 5 monclient: authenticate success, global_id 114264 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: _renew_subs 2024-09-15T20:43:08.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f30f413ba60 con 0x7f30f413c220 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 client.114264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: _renew_subs 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f30f409ea90 con 0x7f30f413c220 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f9fb6640 1 librados: init done 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f30ec001ba0 con 0x7f30f413c220 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f30ec003e90 con 0x7f30f413c220 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:08.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30f1ffb640 4 set_mon_vals no callback set 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.631+0000 7f30ebfff640 10 monclient: dump: 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:08.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30f9fb6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30f9fb6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30f9fb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30d8005090 con 0x7f30f413c220 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f30ec0040b0 con 0x7f30f413c220 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(447..447 src has 223..447) ==== 5354+0+0 (unknown 3060371923 0 0) 0x7f30ec04bb80 con 0x7f30f413c220 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30ebfff640 10 client.114264.objecter ms_dispatch 0x7f30f4000ba0 osd_map(447..447 src has 223..447) 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30ebfff640 3 client.114264.objecter handle_osd_map got epochs [447,447] > 0 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30ebfff640 3 client.114264.objecter handle_osd_map decoding full epoch 447 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30ebfff640 20 client.114264.objecter dump_active .. 0 homeless 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30f37fe640 10 monclient: get_auth_request con 0x7f30d403df50 auth_method 0 2024-09-15T20:43:08.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.635+0000 7f30ebfff640 10 client.114264.objecter ms_handle_connect 0x7f30d403df50 2024-09-15T20:43:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.639+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f30f413f900 con 0x7f30f413c220 2024-09-15T20:43:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.639+0000 7f30ebfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:08.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.639+0000 7f30ebfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:08.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.859+0000 7f30f9fb6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:43:08.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.859+0000 7f30f9fb6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:08.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.859+0000 7f30f9fb6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:08.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.859+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f30d8005380 con 0x7f30f413c220 2024-09-15T20:43:08.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.859+0000 7f30ebfff640 1 -- 172.21.15.78:0/1851155297 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 3840691416) 0x7f30ec016e80 con 0x7f30f413c220 2024-09-15T20:43:08.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.859+0000 7f30ebfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:08.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.859+0000 7f30ebfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:08.862 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:43:08.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 10 librados: watch_flush enter 2024-09-15T20:43:08.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 10 librados: watch_flush exit 2024-09-15T20:43:08.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 20 client.114264.objecter shutdown clearing up homeless session... 2024-09-15T20:43:08.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 10 client.114264.objecter successfully canceled tick 2024-09-15T20:43:08.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 >> v1:172.21.15.146:6812/435598642 conn(0x7f30d403df50 legacy=0x7f30d4040410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 10 monclient: shutdown 2024-09-15T20:43:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 >> v1:172.21.15.78:6789/0 conn(0x7f30f413c220 legacy=0x7f30f41d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 shutdown_connections 2024-09-15T20:43:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 >> 172.21.15.78:0/1851155297 conn(0x7f30f40a63c0 msgr2=0x7f30f40a45b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 shutdown_connections 2024-09-15T20:43:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 1 -- 172.21.15.78:0/1851155297 wait complete. 2024-09-15T20:43:08.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:08.863+0000 7f30f9fb6640 1 librados: shutdown 2024-09-15T20:43:11.880 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:43:12.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient: get_monmap_and_config 2024-09-15T20:43:12.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient: build_initial_monmap 2024-09-15T20:43:12.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient: monmap: 2024-09-15T20:43:12.009 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:12.009 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:12.009 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:12.010070+0000 2024-09-15T20:43:12.009 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:12.010070+0000 2024-09-15T20:43:12.009 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:12.009 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:12.010 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:12.010 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:12.010 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:12.010 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 1 Processor -- start 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 1 -- start start 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient: init 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient(hunting): picked mon.noname-a con 0x7f56f8135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient(hunting): picked mon.noname-c con 0x7f56f80a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient(hunting): picked mon.noname-b con 0x7f56f813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f80745e0 con 0x7f56f8135b60 2024-09-15T20:43:12.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f806ecf0 con 0x7f56f80a24e0 2024-09-15T20:43:12.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f806efd0 con 0x7f56f813c220 2024-09-15T20:43:12.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:12.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56fd386640 10 monclient(hunting): authenticate will time out at 1852.894528s 2024-09-15T20:43:12.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56f6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f56f8135b60 0x7f56f8135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60150/0 (socket says 172.21.15.78:60150) 2024-09-15T20:43:12.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56f77fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f56f813c220 0x7f56f813e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:47858/0 (socket says 172.21.15.78:47858) 2024-09-15T20:43:12.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56f6ffd640 1 -- 172.21.15.78:0/634897760 learned_addr learned my addr 172.21.15.78:0/634897760 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:12.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.007+0000 7f56f67fc640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56f813ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:12.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.211+0000 7f56f67fc640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56f813ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:12.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:12.615+0000 7f56f67fc640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56f813ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:13.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:13.419+0000 7f56f67fc640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56f813ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:15.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 10 monclient(hunting): tick 2024-09-15T20:43:15.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 1 monclient(hunting): continuing hunt 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.78:6790/0 conn(0x7f56f813c220 legacy=0x7f56f813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56f813ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.78:6789/0 conn(0x7f56f8135b60 legacy=0x7f56f8135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f56e40036d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f56f80a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f56f813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f8136650 con 0x7f56e40036d0 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f806ecf0 con 0x7f56f80a24e0 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56e4010400 con 0x7f56f813c220 2024-09-15T20:43:15.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f4ff9640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:15.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.007+0000 7f56f67fc640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e40093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:15.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.211+0000 7f56f67fc640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e40093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:15.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:15.615+0000 7f56f67fc640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e40093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:16.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:16.419+0000 7f56f67fc640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e40093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 10 monclient(hunting): tick 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 1 monclient(hunting): continuing hunt 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.78:6790/0 conn(0x7f56f813c220 legacy=0x7f56e400cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e40093d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.78:6789/0 conn(0x7f56e40036d0 legacy=0x7f56e4005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f56e40106d0 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f56e40129c0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:18.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f56f80a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:18.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56e4011700 con 0x7f56e40129c0 2024-09-15T20:43:18.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f806ecf0 con 0x7f56f80a24e0 2024-09-15T20:43:18.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56e40073c0 con 0x7f56e40106d0 2024-09-15T20:43:18.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f4ff9640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:18.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.007+0000 7f56f77fe640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:18.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.211+0000 7f56f77fe640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:18.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:18.615+0000 7f56f77fe640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e4005b20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 945797413 0 0) 0x7f56e40073c0 con 0x7f56e40106d0 2024-09-15T20:43:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 10 monclient(hunting): my global_id is 124157 2024-09-15T20:43:19.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e0003530 con 0x7f56e40106d0 2024-09-15T20:43:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 499041665 0 0) 0x7f56e4011700 con 0x7f56e40129c0 2024-09-15T20:43:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 10 monclient(hunting): my global_id is 114267 2024-09-15T20:43:19.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.391+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e40073c0 con 0x7f56e40129c0 2024-09-15T20:43:19.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 182180630 0 0) 0x7f56e0003530 con 0x7f56e40106d0 2024-09-15T20:43:19.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.146:6789/0 conn(0x7f56f80a24e0 legacy=0x7f56e4005b20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:43:19.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.78:6789/0 conn(0x7f56e40129c0 legacy=0x7f56e4014db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:43:19.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 1 monclient: found mon.noname-b 2024-09-15T20:43:19.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:43:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e40085f0 con 0x7f56e40106d0 2024-09-15T20:43:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: _finish_auth 0 2024-09-15T20:43:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:43:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:19.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f56e8001600 con 0x7f56e40106d0 2024-09-15T20:43:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:43:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: dump: 2024-09-15T20:43:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.404 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.405 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 5 monclient: authenticate success, global_id 124157 2024-09-15T20:43:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56e80017a0 con 0x7f56e40106d0 2024-09-15T20:43:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 1 -- 172.21.15.78:0/634897760 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f56e8001c30 con 0x7f56e40106d0 2024-09-15T20:43:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:19.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 4 set_mon_vals no callback set 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56f5ffb640 10 monclient: dump: 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.407 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.408 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.408 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 10 monclient: shutdown 2024-09-15T20:43:19.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 1 -- 172.21.15.78:0/634897760 >> v1:172.21.15.78:6790/0 conn(0x7f56e40106d0 legacy=0x7f56e4010ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 1 -- 172.21.15.78:0/634897760 shutdown_connections 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 1 -- 172.21.15.78:0/634897760 >> 172.21.15.78:0/634897760 conn(0x7f56f80a63c0 msgr2=0x7f56f80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 1 -- 172.21.15.78:0/634897760 shutdown_connections 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.399+0000 7f56fd386640 1 -- 172.21.15.78:0/634897760 wait complete. 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: build_initial_monmap 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: monmap: 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.414 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 librados: starting msgr at 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 librados: starting objecter 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 Processor -- start 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 -- start start 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 librados: setting wanted keys 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 librados: calling monclient init 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: init 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:43:19.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient(hunting): picked mon.noname-a con 0x7f56e40106d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient(hunting): picked mon.noname-b con 0x7f56e40129c0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient(hunting): picked mon.noname-c con 0x7f56f80a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f8122180 con 0x7f56e40106d0 2024-09-15T20:43:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f813bfa0 con 0x7f56e40129c0 2024-09-15T20:43:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56f80a5450 con 0x7f56f80a24e0 2024-09-15T20:43:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient(hunting): authenticate will time out at 1860.290230s 2024-09-15T20:43:19.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56f67fc640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7f56e40129c0 legacy=0x7f56f81d1c60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56f6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f56e40106d0 0x7f56f8134420 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60188/0 (socket says 172.21.15.78:60188) 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56f6ffd640 1 -- 172.21.15.78:0/1514527841 learned_addr learned my addr 172.21.15.78:0/1514527841 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 client.?.objecter ms_handle_connect 0x7f56e40106d0 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 client.?.objecter ms_handle_connect 0x7f56f80a24e0 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 33048052 0 0) 0x7f56f8122180 con 0x7f56e40106d0 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): my global_id is 114270 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56e4011600 con 0x7f56e40106d0 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2058584150 0 0) 0x7f56f80a5450 con 0x7f56f80a24e0 2024-09-15T20:43:19.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): my global_id is 124160 2024-09-15T20:43:19.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f8122180 con 0x7f56f80a24e0 2024-09-15T20:43:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3549593021 0 0) 0x7f56e4011600 con 0x7f56e40106d0 2024-09-15T20:43:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56f80a5450 con 0x7f56e40106d0 2024-09-15T20:43:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f56e8001b30 con 0x7f56e40106d0 2024-09-15T20:43:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): dump: 2024-09-15T20:43:19.419 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.420 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.421 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4121349183 0 0) 0x7f56f8122180 con 0x7f56f80a24e0 2024-09-15T20:43:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f56e4011600 con 0x7f56f80a24e0 2024-09-15T20:43:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f56f0001a70 con 0x7f56f80a24e0 2024-09-15T20:43:19.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient(hunting): dump: 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3602637940 0 0) 0x7f56f80a5450 con 0x7f56e40106d0 2024-09-15T20:43:19.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 >> v1:172.21.15.78:6790/0 conn(0x7f56f80a24e0 legacy=0x7f56f81d5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 >> v1:172.21.15.146:6789/0 conn(0x7f56e40129c0 legacy=0x7f56f81d1c60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:43:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:43:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 monclient: found mon.a 2024-09-15T20:43:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56f809d180 con 0x7f56e40106d0 2024-09-15T20:43:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient: _finish_auth 0 2024-09-15T20:43:19.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:43:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 5 monclient: authenticate success, global_id 114270 2024-09-15T20:43:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: _renew_subs 2024-09-15T20:43:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:19.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f56f8121f00 con 0x7f56e40106d0 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 client.114270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: _renew_subs 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f56f811f650 con 0x7f56e40106d0 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56fd386640 1 librados: init done 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f56e8004aa0 con 0x7f56e40106d0 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f56e8004df0 con 0x7f56e40106d0 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:19.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56df7fe640 10 monclient: dump: 2024-09-15T20:43:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.426 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.403+0000 7f56f57fa640 4 set_mon_vals no callback set 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f56e8010e60 con 0x7f56e40106d0 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56fd386640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56fd386640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56fd386640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:19.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56f811f990 con 0x7f56e40106d0 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(448..448 src has 223..448) ==== 5380+0+0 (unknown 4254541579 0 0) 0x7f56e804b510 con 0x7f56e40106d0 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56df7fe640 10 client.114270.objecter ms_dispatch 0x7f56f8000ba0 osd_map(448..448 src has 223..448) 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56df7fe640 3 client.114270.objecter handle_osd_map got epochs [448,448] > 0 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56df7fe640 3 client.114270.objecter handle_osd_map decoding full epoch 448 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56df7fe640 20 client.114270.objecter dump_active .. 0 homeless 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56f67fc640 10 monclient: get_auth_request con 0x7f56e403f6e0 auth_method 0 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.407+0000 7f56df7fe640 10 client.114270.objecter ms_handle_connect 0x7f56e403f6e0 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.411+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f56f8121f00 con 0x7f56e40106d0 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.411+0000 7f56df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:19.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.411+0000 7f56df7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:19.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56fd386640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:43:19.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56fd386640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:19.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56fd386640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:19.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f56f8076db0 con 0x7f56e40106d0 2024-09-15T20:43:19.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56df7fe640 1 -- 172.21.15.78:0/1514527841 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 3238118594) 0x7f56e80168b0 con 0x7f56e40106d0 2024-09-15T20:43:19.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:43:19.669 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56df7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:19.669 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":48,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:43:19.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56fd386640 10 librados: watch_flush enter 2024-09-15T20:43:19.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56fd386640 10 librados: watch_flush exit 2024-09-15T20:43:19.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56fd386640 20 client.114270.objecter shutdown clearing up homeless session... 2024-09-15T20:43:19.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.667+0000 7f56fd386640 10 client.114270.objecter successfully canceled tick 2024-09-15T20:43:19.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 >> v1:172.21.15.146:6812/435598642 conn(0x7f56e403f6e0 legacy=0x7f56e4041ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 10 monclient: shutdown 2024-09-15T20:43:19.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:19.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 >> v1:172.21.15.78:6789/0 conn(0x7f56e40106d0 legacy=0x7f56f8134420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 shutdown_connections 2024-09-15T20:43:19.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 >> 172.21.15.78:0/1514527841 conn(0x7f56f80a63c0 msgr2=0x7f56f809e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:19.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 shutdown_connections 2024-09-15T20:43:19.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 1 -- 172.21.15.78:0/1514527841 wait complete. 2024-09-15T20:43:19.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.671+0000 7f56fd386640 1 librados: shutdown 2024-09-15T20:43:19.688 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:43:19.688 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T20:43:19.688 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:43:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient: get_monmap_and_config 2024-09-15T20:43:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient: build_initial_monmap 2024-09-15T20:43:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient: monmap: 2024-09-15T20:43:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.808 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:19.809873+0000 2024-09-15T20:43:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:19.809873+0000 2024-09-15T20:43:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:19.809 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:19.809 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 1 Processor -- start 2024-09-15T20:43:19.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 1 -- start start 2024-09-15T20:43:19.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient: init 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f86b8074630 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f86b809d950 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f86b8074a10 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b8134bc0 con 0x7f86b8074a10 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b809f900 con 0x7f86b8074630 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b809fbe0 con 0x7f86b809d950 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient(hunting): authenticate will time out at 1860.694631s 2024-09-15T20:43:19.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bda8e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f86b8074a10 0x7f86b813d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60196/0 (socket says 172.21.15.78:60196) 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bda8e640 1 -- 172.21.15.78:0/996177550 learned_addr learned my addr 172.21.15.78:0/996177550 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bd28d640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7f86b8074630 legacy=0x7f86b8133da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bca8c640 1 --1- 172.21.15.78:0/996177550 >> v1:172.21.15.78:6790/0 conn(0x7f86b809d950 0x7f86b81344b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:47886/0 (socket says 172.21.15.78:47886) 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795470913 0 0) 0x7f86b8134bc0 con 0x7f86b8074a10 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient(hunting): my global_id is 114273 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86a4003530 con 0x7f86b8074a10 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2336502042 0 0) 0x7f86b809fbe0 con 0x7f86b809d950 2024-09-15T20:43:19.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient(hunting): my global_id is 124163 2024-09-15T20:43:19.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86b8134bc0 con 0x7f86b809d950 2024-09-15T20:43:19.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2955582264 0 0) 0x7f86b8134bc0 con 0x7f86b809d950 2024-09-15T20:43:19.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 >> v1:172.21.15.146:6789/0 conn(0x7f86b8074630 legacy=0x7f86b8133da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:43:19.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 >> v1:172.21.15.78:6789/0 conn(0x7f86b8074a10 legacy=0x7f86b813d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: _finish_hunting 0 2024-09-15T20:43:19.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 monclient: found mon.noname-b 2024-09-15T20:43:19.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:19.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:43:19.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b809fec0 con 0x7f86b809d950 2024-09-15T20:43:19.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: _finish_auth 0 2024-09-15T20:43:19.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: _check_auth_tickets 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86a80015b0 con 0x7f86b809d950 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 5 monclient: authenticate success, global_id 124163 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: dump: 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.823 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86a8001750 con 0x7f86b809d950 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 1 -- 172.21.15.78:0/996177550 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86a8001b60 con 0x7f86b809d950 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86bf4f1640 4 set_mon_vals no callback set 2024-09-15T20:43:19.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.807+0000 7f86affff640 10 monclient: dump: 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: shutdown 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:19.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- 172.21.15.78:0/996177550 >> v1:172.21.15.78:6790/0 conn(0x7f86b809d950 legacy=0x7f86b81344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- 172.21.15.78:0/996177550 shutdown_connections 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- 172.21.15.78:0/996177550 >> 172.21.15.78:0/996177550 conn(0x7f86b80a63c0 msgr2=0x7f86b80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- 172.21.15.78:0/996177550 shutdown_connections 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- 172.21.15.78:0/996177550 wait complete. 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: build_initial_monmap 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: monmap: 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.826 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 librados: starting msgr at 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 librados: starting objecter 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 Processor -- start 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- start start 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 librados: setting wanted keys 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 librados: calling monclient init 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: init 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient(hunting): picked mon.noname-a con 0x7f86b8074630 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient(hunting): picked mon.noname-c con 0x7f86b8074a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:19.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient(hunting): picked mon.noname-b con 0x7f86b809d950 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b81c9130 con 0x7f86b8074630 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b813a580 con 0x7f86b809d950 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f86b81301f0 con 0x7f86b8074a10 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient(hunting): authenticate will time out at 1860.697759s 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bd28d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f86b8074630 0x7f86b812f710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60202/0 (socket says 172.21.15.78:60202) 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bca8c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f86b8074a10 0x7f86b81d1970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:47896/0 (socket says 172.21.15.78:47896) 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bd28d640 1 -- 172.21.15.78:0/1832194850 learned_addr learned my addr 172.21.15.78:0/1832194850 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bda8e640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.146:6789/0 conn(0x7f86b809d950 legacy=0x7f86b81d50a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 client.?.objecter ms_handle_connect 0x7f86b8074630 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 client.?.objecter ms_handle_connect 0x7f86b8074a10 2024-09-15T20:43:19.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4209123417 0 0) 0x7f86b0003e60 con 0x7f86b8074630 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): my global_id is 114276 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f869c004ec0 con 0x7f86b8074630 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1772586779 0 0) 0x7f86b81301f0 con 0x7f86b8074a10 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): my global_id is 124166 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86b0003e60 con 0x7f86b8074a10 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1033040415 0 0) 0x7f869c004ec0 con 0x7f86b8074630 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86b81301f0 con 0x7f86b8074630 2024-09-15T20:43:19.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86b00047b0 con 0x7f86b8074630 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): dump: 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4291119473 0 0) 0x7f86b0003e60 con 0x7f86b8074a10 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f869c004ec0 con 0x7f86b8074a10 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86a8001470 con 0x7f86b8074a10 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient(hunting): dump: 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.831 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2164687546 0 0) 0x7f86b81301f0 con 0x7f86b8074630 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.78:6790/0 conn(0x7f86b8074a10 legacy=0x7f86b81d1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.146:6789/0 conn(0x7f86b809d950 legacy=0x7f86b81d50a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient: _finish_hunting 0 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 monclient: found mon.a 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b813a860 con 0x7f86b8074630 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient: _finish_auth 0 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient: _check_auth_tickets 2024-09-15T20:43:19.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 5 monclient: authenticate success, global_id 114276 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: _renew_subs 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f86b813a2a0 con 0x7f86b8074630 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 client.114276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: _renew_subs 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f86b812fe20 con 0x7f86b8074630 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 librados: init done 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f86b0004950 con 0x7f86b8074630 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86b0004d20 con 0x7f86b8074630 2024-09-15T20:43:19.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 monclient: dump: 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86af7fe640 4 set_mon_vals no callback set 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f86b0004f80 con 0x7f86b8074630 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86b80008e0 con 0x7f86b802e5d0 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(448..448 src has 223..448) ==== 5380+0+0 (unknown 4254541579 0 0) 0x7f86b004bc70 con 0x7f86b8074630 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 client.114276.objecter ms_dispatch 0x7f86b8000ba0 osd_map(448..448 src has 223..448) 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 3 client.114276.objecter handle_osd_map got epochs [448,448] > 0 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 3 client.114276.objecter handle_osd_map decoding full epoch 448 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 20 client.114276.objecter dump_active .. 0 homeless 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 client.114276.objecter ms_handle_connect 0x7f86b802e5d0 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.811+0000 7f86adffb640 10 client.114276.objecter resend_mon_ops 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2805866188 0 0) 0x7f86b80008e0 con 0x7f86b802e5d0 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient(hunting): my global_id is 114279 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86b81301f0 con 0x7f86b802e5d0 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86bca8c640 10 monclient: get_auth_request con 0x7f869c03e040 auth_method 0 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2884218288 0 0) 0x7f86b81301f0 con 0x7f86b802e5d0 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86b80008e0 con 0x7f86b802e5d0 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86b4001920 con 0x7f86b802e5d0 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient: dump: 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.836 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2887605879 0 0) 0x7f86b80008e0 con 0x7f86b802e5d0 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f86b802e9b0 con 0x7f86b802e5d0 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 client.114276.objecter ms_handle_connect 0x7f869c03e040 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f86b4001630 con 0x7f86b802e5d0 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.78:6789/0 conn(0x7f86b802e5d0 legacy=0x7f86b8063e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.815+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.78:6789/0 conn(0x7f86b802e5d0 legacy=0x7f86b8063e00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86bf4f1640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:43:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f86b80711e0 con 0x7f86b81c86d0 2024-09-15T20:43:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 client.114276.objecter ms_handle_connect 0x7f86b81c86d0 2024-09-15T20:43:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 client.114276.objecter resend_mon_ops 2024-09-15T20:43:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4177459510 0 0) 0x7f86b80711e0 con 0x7f86b81c86d0 2024-09-15T20:43:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient(hunting): my global_id is 114282 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f86b80008e0 con 0x7f86b81c86d0 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1312438185 0 0) 0x7f86b80008e0 con 0x7f86b81c86d0 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f86b80711e0 con 0x7f86b81c86d0 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f86b4004a50 con 0x7f86b81c86d0 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient: dump: 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.840 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3477380827 0 0) 0x7f86b80711e0 con 0x7f86b81c86d0 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f86b81c8ab0 con 0x7f86b81c86d0 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 1403768442) 0x7f86b4004b30 con 0x7f86b81c86d0 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.78:6789/0 conn(0x7f86b81c86d0 legacy=0x7f86b8070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86adffb640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.78:6789/0 conn(0x7f86b81c86d0 legacy=0x7f86b8070ad0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:19.841 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 48, 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 125662, 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:43:19.842 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:43:19.843 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:19.844 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:43:19.845 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:19.846 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:19.847 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.848 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:43:19.849 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:19.850 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:19.851 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:43:19.852 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:19.853 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:19.854 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:19.854 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:43:19.854 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:43:19.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86bf4f1640 10 librados: watch_flush enter 2024-09-15T20:43:19.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86bf4f1640 10 librados: watch_flush exit 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86bf4f1640 20 client.114276.objecter shutdown clearing up homeless session... 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.819+0000 7f86bf4f1640 10 client.114276.objecter successfully canceled tick 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.146:6812/435598642 conn(0x7f869c03e040 legacy=0x7f869c040500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 10 monclient: shutdown 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 >> v1:172.21.15.78:6789/0 conn(0x7f86b8074630 legacy=0x7f86b812f710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bda8e640 1 -- 172.21.15.78:0/1832194850 reap_dead start 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bda8e640 1 -- 172.21.15.78:0/1832194850 reap_dead start 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 shutdown_connections 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 >> 172.21.15.78:0/1832194850 conn(0x7f86b80a63c0 msgr2=0x7f86b806e3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 shutdown_connections 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 1 -- 172.21.15.78:0/1832194850 wait complete. 2024-09-15T20:43:19.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.823+0000 7f86bf4f1640 1 librados: shutdown 2024-09-15T20:43:19.857 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:43:19.857 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T20:43:19.857 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:43:19.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: get_monmap_and_config 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: build_initial_monmap 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: monmap: 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:43:19.973184+0000 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:43:19.973184+0000 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:19.972 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:43:19.973 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:43:19.973 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 Processor -- start 2024-09-15T20:43:19.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- start start 2024-09-15T20:43:19.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: init 2024-09-15T20:43:19.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:19.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:43:19.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient(hunting): picked mon.noname-c con 0x7f298c134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:19.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient(hunting): picked mon.noname-a con 0x7f298c09f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:19.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient(hunting): picked mon.noname-b con 0x7f298c06e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:19.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298c0a0470 con 0x7f298c09f5a0 2024-09-15T20:43:19.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298c09feb0 con 0x7f298c134bb0 2024-09-15T20:43:19.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298c09e4a0 con 0x7f298c06e960 2024-09-15T20:43:19.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:19.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient(hunting): authenticate will time out at 1860.857696s 2024-09-15T20:43:19.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298b7fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7f298c134bb0 legacy=0x7f298c134f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:19.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298bfff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f298c06e960 0x7f298c13d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:47898/0 (socket says 172.21.15.78:47898) 2024-09-15T20:43:19.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298bfff640 1 -- 172.21.15.78:0/2719824340 learned_addr learned my addr 172.21.15.78:0/2719824340 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:19.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298affd640 1 --1- 172.21.15.78:0/2719824340 >> v1:172.21.15.78:6789/0 conn(0x7f298c09f5a0 0x7f298c1399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:60218/0 (socket says 172.21.15.78:60218) 2024-09-15T20:43:19.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1806544225 0 0) 0x7f2980001120 con 0x7f298c09f5a0 2024-09-15T20:43:19.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient(hunting): my global_id is 114285 2024-09-15T20:43:19.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2974003530 con 0x7f298c09f5a0 2024-09-15T20:43:19.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 565981488 0 0) 0x7f298c09e4a0 con 0x7f298c06e960 2024-09-15T20:43:19.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:43:19.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:19.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:19.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient(hunting): my global_id is 124169 2024-09-15T20:43:19.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2980001120 con 0x7f298c06e960 2024-09-15T20:43:19.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 825308072 0 0) 0x7f2980001120 con 0x7f298c06e960 2024-09-15T20:43:19.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 >> v1:172.21.15.146:6789/0 conn(0x7f298c134bb0 legacy=0x7f298c134f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:43:19.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 >> v1:172.21.15.78:6789/0 conn(0x7f298c09f5a0 legacy=0x7f298c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:43:19.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 monclient: found mon.noname-b 2024-09-15T20:43:19.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:19.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:43:19.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c0a0ff0 con 0x7f298c06e960 2024-09-15T20:43:19.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: _finish_auth 0 2024-09-15T20:43:19.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:43:19.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:19.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2984001a00 con 0x7f298c06e960 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: dump: 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.994 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 5 monclient: authenticate success, global_id 124169 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2984001ba0 con 0x7f298c06e960 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 1 -- 172.21.15.78:0/2719824340 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2984002030 con 0x7f298c06e960 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f298a7fc640 10 monclient: dump: 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:19.995 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: get_monmap_and_config success 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 4 set_mon_vals no callback set 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: shutdown 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- 172.21.15.78:0/2719824340 >> v1:172.21.15.78:6790/0 conn(0x7f298c06e960 legacy=0x7f298c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- 172.21.15.78:0/2719824340 shutdown_connections 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- 172.21.15.78:0/2719824340 >> 172.21.15.78:0/2719824340 conn(0x7f298c0a63c0 msgr2=0x7f298c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:19.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- 172.21.15.78:0/2719824340 shutdown_connections 2024-09-15T20:43:19.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 1 -- 172.21.15.78:0/2719824340 wait complete. 2024-09-15T20:43:19.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: build_initial_monmap 2024-09-15T20:43:19.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.971+0000 7f2992129640 10 monclient: monmap: 2024-09-15T20:43:19.997 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:43:19.997 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:43:19.997 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:43:19.997 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:43:19.997 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 librados: starting msgr at 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 librados: starting objecter 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 Processor -- start 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 -- start start 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 librados: setting wanted keys 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 librados: calling monclient init 2024-09-15T20:43:19.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient: init 2024-09-15T20:43:19.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient: _reopen_session rank -1 2024-09-15T20:43:19.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:43:19.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient(hunting): picked mon.noname-b con 0x7f298c06e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:43:19.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient(hunting): picked mon.noname-a con 0x7f298c09f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:43:19.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient(hunting): picked mon.noname-c con 0x7f298c134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:43:20.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298c0a4dc0 con 0x7f298c09f5a0 2024-09-15T20:43:20.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298c09d180 con 0x7f298c06e960 2024-09-15T20:43:20.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f298c123610 con 0x7f298c134bb0 2024-09-15T20:43:20.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient(hunting): _renew_subs 2024-09-15T20:43:20.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient(hunting): authenticate will time out at 1860.860937s 2024-09-15T20:43:20.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f298bfff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f298c134bb0 0x7f298c1d4a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:47912/0 (socket says 172.21.15.78:47912) 2024-09-15T20:43:20.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f298bfff640 1 -- 172.21.15.78:0/918983541 learned_addr learned my addr 172.21.15.78:0/918983541 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:43:20.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f298b7fe640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.146:6789/0 conn(0x7f298c06e960 legacy=0x7f298c1cc6f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:43:20.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 client.?.objecter ms_handle_connect 0x7f298c134bb0 2024-09-15T20:43:20.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:20.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 client.?.objecter ms_handle_connect 0x7f298c09f5a0 2024-09-15T20:43:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:43:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2396622898 0 0) 0x7f298c123610 con 0x7f298c134bb0 2024-09-15T20:43:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): my global_id is 124172 2024-09-15T20:43:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2978004ec0 con 0x7f298c134bb0 2024-09-15T20:43:20.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3490514821 0 0) 0x7f2980001b70 con 0x7f298c09f5a0 2024-09-15T20:43:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): my global_id is 114288 2024-09-15T20:43:20.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f298c123610 con 0x7f298c09f5a0 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2984929440 0 0) 0x7f2978004ec0 con 0x7f298c134bb0 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2980001b70 con 0x7f298c134bb0 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2984001f30 con 0x7f298c134bb0 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): dump: 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.004 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2623495851 0 0) 0x7f298c123610 con 0x7f298c09f5a0 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2978004ec0 con 0x7f298c09f5a0 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f29800015f0 con 0x7f298c09f5a0 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): dump: 2024-09-15T20:43:20.005 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4068645466 0 0) 0x7f2980001b70 con 0x7f298c134bb0 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.146:6789/0 conn(0x7f298c06e960 legacy=0x7f298c1cc6f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.78:6789/0 conn(0x7f298c09f5a0 legacy=0x7f298c132d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: _finish_hunting 0 2024-09-15T20:43:20.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 monclient: found mon.c 2024-09-15T20:43:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:43:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f298c124730 con 0x7f298c134bb0 2024-09-15T20:43:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: _finish_auth 0 2024-09-15T20:43:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: _check_auth_tickets 2024-09-15T20:43:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:43:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 5 monclient: authenticate success, global_id 124172 2024-09-15T20:43:20.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient: _renew_subs 2024-09-15T20:43:20.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:20.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f298c122f30 con 0x7f298c134bb0 2024-09-15T20:43:20.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 client.124172.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:43:20.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient: _renew_subs 2024-09-15T20:43:20.008 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f298c133610 con 0x7f298c134bb0 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 librados: init done 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2984001b80 con 0x7f298c134bb0 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2984005120 con 0x7f298c134bb0 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2989ffb640 4 set_mon_vals no callback set 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: dump: 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.009 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.010 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:20.010 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:20.010 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:20.010 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:20.010 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:20.010 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:20.010 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:20.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f296c0065d0 con 0x7f296c002870 2024-09-15T20:43:20.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 client.124172.objecter ms_handle_connect 0x7f296c002870 2024-09-15T20:43:20.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 client.124172.objecter resend_mon_ops 2024-09-15T20:43:20.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f2984011190 con 0x7f298c134bb0 2024-09-15T20:43:20.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(448..448 src has 223..448) ==== 5380+0+0 (unknown 4254541579 0 0) 0x7f298404bb90 con 0x7f298c134bb0 2024-09-15T20:43:20.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 client.124172.objecter ms_dispatch 0x7f298c000ba0 osd_map(448..448 src has 223..448) 2024-09-15T20:43:20.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 3 client.124172.objecter handle_osd_map got epochs [448,448] > 0 2024-09-15T20:43:20.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 3 client.124172.objecter handle_osd_map decoding full epoch 448 2024-09-15T20:43:20.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 20 client.124172.objecter dump_active .. 0 homeless 2024-09-15T20:43:20.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 274486770 0 0) 0x7f296c0065d0 con 0x7f296c002870 2024-09-15T20:43:20.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:20.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:20.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:20.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient(hunting): my global_id is 124175 2024-09-15T20:43:20.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2980001b70 con 0x7f296c002870 2024-09-15T20:43:20.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1129377056 0 0) 0x7f2980001b70 con 0x7f296c002870 2024-09-15T20:43:20.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f296c0065d0 con 0x7f296c002870 2024-09-15T20:43:20.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f297c003f30 con 0x7f296c002870 2024-09-15T20:43:20.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:20.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:20.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: dump: 2024-09-15T20:43:20.015 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:20.015 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:20.015 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.015 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:20.016 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:20.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f298affd640 10 monclient: get_auth_request con 0x7f297803dfe0 auth_method 0 2024-09-15T20:43:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2042299505 0 0) 0x7f296c0065d0 con 0x7f296c002870 2024-09-15T20:43:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f296c00a170 con 0x7f296c002870 2024-09-15T20:43:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 client.124172.objecter ms_handle_connect 0x7f297803dfe0 2024-09-15T20:43:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f297c0054b0 con 0x7f296c002870 2024-09-15T20:43:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:43:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:43:20.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.78:6790/0 conn(0x7f296c002870 legacy=0x7f296c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.975+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.78:6790/0 conn(0x7f296c002870 legacy=0x7f296c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:43:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f296c009e80 con 0x7f296c009140 2024-09-15T20:43:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 client.124172.objecter ms_handle_connect 0x7f296c009140 2024-09-15T20:43:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 client.124172.objecter resend_mon_ops 2024-09-15T20:43:20.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 403057080 0 0) 0x7f296c009e80 con 0x7f296c009140 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient(hunting): _negotiate 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient(hunting): my global_id is 124178 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f296c0065d0 con 0x7f296c009140 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3314817206 0 0) 0x7f296c0065d0 con 0x7f296c009140 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f296c009e80 con 0x7f296c009140 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f297c0042a0 con 0x7f296c009140 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:43:20.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient: dump: 2024-09-15T20:43:20.020 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:43:20.020 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:43:20.020 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.020 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:43:20.020 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:43:20.021 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:43:20.021 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:43:20.021 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:43:20.021 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:43:20.021 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:43:20.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 536366792 0 0) 0x7f296c009e80 con 0x7f296c009140 2024-09-15T20:43:20.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f296c00c420 con 0x7f296c009140 2024-09-15T20:43:20.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3959 (unknown 0 0 955519531) 0x7f297c005740 con 0x7f296c009140 2024-09-15T20:43:20.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:43:20.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:43:20.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.78:6790/0 conn(0x7f296c009140 legacy=0x7f296c0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:20.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2973fff640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.78:6790/0 conn(0x7f296c009140 legacy=0x7f296c0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:43:20.023 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:43:20.023 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:20.023 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:20.023 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:43:20.023 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 48, 2024-09-15T20:43:20.023 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17911, 2024-09-15T20:43:20.024 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:43:20.024 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:43:20.024 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:43:20.024 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:20.024 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:43:20.024 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:20.025 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:43:20.025 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:43:20.025 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:20.025 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:20.025 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:20.025 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:20.026 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:20.027 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:20.027 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:20.027 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:43:20.027 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:43:20.027 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:20.027 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:20.027 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:20.028 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:20.028 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:20.028 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:20.028 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:20.028 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:20.028 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:20.028 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:20.029 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:20.029 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:20.029 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.029 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:43:20.029 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:43:20.029 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:43:20.029 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:43:20.029 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:43:20.030 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:43:20.030 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:20.030 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:43:20.030 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:43:20.030 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:43:20.030 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:43:20.030 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:43:20.031 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:43:20.031 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:43:20.031 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:43:20.031 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:43:20.031 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:43:20.031 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:43:20.032 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:43:20.032 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:43:20.032 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:43:20.032 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:43:20.032 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:43:20.032 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:43:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:43:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:43:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:43:20.033 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:43:20.034 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:20.034 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:43:20.034 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.034 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:43:20.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:20.035 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:43:20.035 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:43:20.035 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:20.035 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:20.035 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:20.036 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:20.036 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:43:20.036 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:20.036 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:20.036 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:20.037 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.037 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:43:20.037 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:43:20.037 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:20.037 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:43:20.038 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.039 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:43:20.040 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:43:20.041 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:43:20.042 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 10 librados: watch_flush enter 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 10 librados: watch_flush exit 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 20 client.124172.objecter shutdown clearing up homeless session... 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 10 client.124172.objecter successfully canceled tick 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.146:6812/435598642 conn(0x7f297803dfe0 legacy=0x7f29780404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 10 monclient: shutdown 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 >> v1:172.21.15.78:6790/0 conn(0x7f298c134bb0 legacy=0x7f298c1d4a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f298bfff640 1 -- 172.21.15.78:0/918983541 reap_dead start 2024-09-15T20:43:20.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f298bfff640 1 -- 172.21.15.78:0/918983541 reap_dead start 2024-09-15T20:43:20.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 shutdown_connections 2024-09-15T20:43:20.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 >> 172.21.15.78:0/918983541 conn(0x7f298c0a63c0 msgr2=0x7f298c1339e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:43:20.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 shutdown_connections 2024-09-15T20:43:20.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 1 -- 172.21.15.78:0/918983541 wait complete. 2024-09-15T20:43:20.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:43:19.983+0000 7f2992129640 1 librados: shutdown 2024-09-15T20:43:20.046 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:43:20.046 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:44:50.047 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T20:44:50.047 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T20:44:50.048 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T20:44:50.051 INFO:tasks.ceph.mon.b:Started 2024-09-15T20:44:50.051 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:44:50.051 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:44:50.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient: get_monmap_and_config 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient: build_initial_monmap 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient: monmap: 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:50.181585+0000 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:50.181585+0000 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:50.180 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 1 Processor -- start 2024-09-15T20:44:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 1 -- start start 2024-09-15T20:44:50.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient: init 2024-09-15T20:44:50.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:50.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:44:50.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient(hunting): picked mon.noname-b con 0x7fd1cc074110 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:50.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient(hunting): picked mon.noname-a con 0x7fd1cc074c00 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient(hunting): picked mon.noname-c con 0x7fd1cc13e6d0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1cc0a1d90 con 0x7fd1cc074c00 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1cc0a1ab0 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1cc0a2070 con 0x7fd1cc074110 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1d1b5a640 10 monclient(hunting): authenticate will time out at 1951.067033s 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1cb7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd1cc074110 0x7fd1cc0744f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56454/0 (socket says 172.21.15.78:56454) 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1cb7fe640 1 -- 172.21.15.78:0/3705029139 learned_addr learned my addr 172.21.15.78:0/3705029139 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3233716021 0 0) 0x7fd1b4001120 con 0x7fd1cc074c00 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient(hunting): my global_id is 114315 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b8003530 con 0x7fd1cc074c00 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2915723443 0 0) 0x7fd1cc0a2070 con 0x7fd1cc074110 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient(hunting): my global_id is 124202 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b4001120 con 0x7fd1cc074110 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 159766766 0 0) 0x7fd1b8003530 con 0x7fd1cc074c00 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 >> v1:172.21.15.78:6790/0 conn(0x7fd1cc074110 legacy=0x7fd1cc0744f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 >> v1:172.21.15.146:6789/0 conn(0x7fd1cc13e6d0 legacy=0x7fd1cc140ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 10 monclient: _finish_hunting 0 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.178+0000 7fd1caffd640 1 monclient: found mon.noname-a 2024-09-15T20:44:50.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1cc09df30 con 0x7fd1cc074c00 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: _finish_auth 0 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: _check_auth_tickets 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd1b4001870 con 0x7fd1cc074c00 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: dump: 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.186 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 5 monclient: authenticate success, global_id 114315 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1b4001a90 con 0x7fd1cc074c00 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 4 set_mon_vals no callback set 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 1 -- 172.21.15.78:0/3705029139 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd1b4003600 con 0x7fd1cc074c00 2024-09-15T20:44:50.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1caffd640 10 monclient: dump: 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: shutdown 2024-09-15T20:44:50.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:50.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/3705029139 >> v1:172.21.15.78:6789/0 conn(0x7fd1cc074c00 legacy=0x7fd1cc13cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/3705029139 shutdown_connections 2024-09-15T20:44:50.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/3705029139 >> 172.21.15.78:0/3705029139 conn(0x7fd1cc0a3f10 msgr2=0x7fd1cc0a4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:50.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/3705029139 shutdown_connections 2024-09-15T20:44:50.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/3705029139 wait complete. 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: build_initial_monmap 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: monmap: 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:50.194 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 librados: starting msgr at 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 librados: starting objecter 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 Processor -- start 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- start start 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 librados: setting wanted keys 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 librados: calling monclient init 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: init 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient(hunting): picked mon.noname-b con 0x7fd1cc074110 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient(hunting): picked mon.noname-c con 0x7fd1cc074c00 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient(hunting): picked mon.noname-a con 0x7fd1cc13e6d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1cc1cd840 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1cc09e200 con 0x7fd1cc074110 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1cc11eec0 con 0x7fd1cc074c00 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient(hunting): authenticate will time out at 1951.070484s 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd1cc074c00 0x7fd1cc1d1c50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56456/0 (socket says 172.21.15.78:56456) 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1cbfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fd1cc13e6d0 0x7fd1cc1d5380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57752/0 (socket says 172.21.15.78:57752) 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c2ffd640 1 -- 172.21.15.78:0/57647693 learned_addr learned my addr 172.21.15.78:0/57647693 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1cc13e6d0 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1cc074c00 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 client.?.objecter ms_handle_connect 0x7fd1cc074110 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3947949847 0 0) 0x7fd1cc0a1ab0 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): my global_id is 114318 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1ac004ec0 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1198648186 0 0) 0x7fd1cc11eec0 con 0x7fd1cc074c00 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): my global_id is 124205 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1cc0a1ab0 con 0x7fd1cc074c00 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3173232550 0 0) 0x7fd1ac004ec0 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1cc11eec0 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd1c4001c40 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): dump: 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3888793896 0 0) 0x7fd1cc0a1ab0 con 0x7fd1cc074c00 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd1ac004ec0 con 0x7fd1cc074c00 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd1b4001c40 con 0x7fd1cc074c00 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient(hunting): dump: 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.199 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2973804353 0 0) 0x7fd1cc11eec0 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 >> v1:172.21.15.78:6790/0 conn(0x7fd1cc074c00 legacy=0x7fd1cc1d1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 >> v1:172.21.15.146:6789/0 conn(0x7fd1cc074110 legacy=0x7fd1cc135880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 monclient: found mon.a 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1cc121220 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient: _finish_auth 0 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:44:50.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 5 monclient: authenticate success, global_id 114318 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: _renew_subs 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd1cc120980 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 client.114318.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: _renew_subs 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fd1cc11e670 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1d1b5a640 1 librados: init done 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd1c40039a0 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd1c4003d70 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c9ffb640 4 set_mon_vals no callback set 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.182+0000 7fd1c8ff9640 10 monclient: dump: 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1d1b5a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1d1b5a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1d1b5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b0005090 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd1c4003f10 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7fd1c404bb80 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1c8ff9640 10 client.114318.objecter ms_dispatch 0x7fd1cc000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1c8ff9640 3 client.114318.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1c8ff9640 3 client.114318.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1c8ff9640 20 client.114318.objecter dump_active .. 0 homeless 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1cb7fe640 10 monclient: get_auth_request con 0x7fd1ac03e060 auth_method 0 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.186+0000 7fd1c8ff9640 10 client.114318.objecter ms_handle_connect 0x7fd1ac03e060 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.190+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd1cc1cd840 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.190+0000 7fd1c8ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.190+0000 7fd1c8ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:50.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.430+0000 7fd1d1b5a640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:44:50.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.430+0000 7fd1d1b5a640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:44:50.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd1b0005380 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1c8ff9640 1 -- 172.21.15.78:0/57647693 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 1707786731) 0x7fd1c4016c80 con 0x7fd1cc13e6d0 2024-09-15T20:44:50.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1c8ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:44:50.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1c8ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:50.436 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:44:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 10 librados: watch_flush enter 2024-09-15T20:44:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 10 librados: watch_flush exit 2024-09-15T20:44:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 20 client.114318.objecter shutdown clearing up homeless session... 2024-09-15T20:44:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 10 client.114318.objecter successfully canceled tick 2024-09-15T20:44:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 >> v1:172.21.15.146:6812/435598642 conn(0x7fd1ac03e060 legacy=0x7fd1ac040520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 10 monclient: shutdown 2024-09-15T20:44:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 >> v1:172.21.15.78:6789/0 conn(0x7fd1cc13e6d0 legacy=0x7fd1cc1d5380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 shutdown_connections 2024-09-15T20:44:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 >> 172.21.15.78:0/57647693 conn(0x7fd1cc0a3f10 msgr2=0x7fd1cc0a0680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 shutdown_connections 2024-09-15T20:44:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 1 -- 172.21.15.78:0/57647693 wait complete. 2024-09-15T20:44:50.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.434+0000 7fd1d1b5a640 1 librados: shutdown 2024-09-15T20:44:50.454 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:44:50.454 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:44:50.454 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:44:50.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: get_monmap_and_config 2024-09-15T20:44:50.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: build_initial_monmap 2024-09-15T20:44:50.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: monmap: 2024-09-15T20:44:50.575 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:50.575 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.576 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:50.577272+0000 2024-09-15T20:44:50.576 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:50.577272+0000 2024-09-15T20:44:50.576 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:50.576 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.576 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:50.576 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:50.576 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:50.576 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 Processor -- start 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- start start 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: init 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient(hunting): picked mon.noname-b con 0x7f699c134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient(hunting): picked mon.noname-c con 0x7f699c09f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient(hunting): picked mon.noname-a con 0x7f699c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699c0a0470 con 0x7f699c06e960 2024-09-15T20:44:50.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699c09feb0 con 0x7f699c09f5a0 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699c09e4a0 con 0x7f699c134bb0 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient(hunting): authenticate will time out at 1951.461790s 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a2101640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f699c134bb0 0x7f699c134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56464/0 (socket says 172.21.15.78:56464) 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a2101640 1 -- 172.21.15.78:0/529400176 learned_addr learned my addr 172.21.15.78:0/529400176 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2780299412 0 0) 0x7f699c09e4a0 con 0x7f699c134bb0 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient(hunting): my global_id is 124208 2024-09-15T20:44:50.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6988003530 con 0x7f699c134bb0 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 287539468 0 0) 0x7f69940014a0 con 0x7f699c06e960 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient(hunting): my global_id is 114321 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f699c09e4a0 con 0x7f699c06e960 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2616176306 0 0) 0x7f6988003530 con 0x7f699c134bb0 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 >> v1:172.21.15.146:6789/0 conn(0x7f699c09f5a0 legacy=0x7f699c1399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 >> v1:172.21.15.78:6789/0 conn(0x7f699c06e960 legacy=0x7f699c13d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: _finish_hunting 0 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 monclient: found mon.noname-b 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:50.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699c0a0ff0 con 0x7f699c134bb0 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: _finish_auth 0 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: _check_auth_tickets 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f698c0016a0 con 0x7f699c134bb0 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 5 monclient: authenticate success, global_id 124208 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: dump: 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.590 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f698c0018c0 con 0x7f699c134bb0 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 1 -- 172.21.15.78:0/529400176 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f698c001d10 con 0x7f699c134bb0 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:50.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a10ff640 10 monclient: dump: 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 4 set_mon_vals no callback set 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: shutdown 2024-09-15T20:44:50.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- 172.21.15.78:0/529400176 >> v1:172.21.15.78:6790/0 conn(0x7f699c134bb0 legacy=0x7f699c134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- 172.21.15.78:0/529400176 shutdown_connections 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- 172.21.15.78:0/529400176 >> 172.21.15.78:0/529400176 conn(0x7f699c0a63c0 msgr2=0x7f699c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- 172.21.15.78:0/529400176 shutdown_connections 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 1 -- 172.21.15.78:0/529400176 wait complete. 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: build_initial_monmap 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.574+0000 7f69a4365640 10 monclient: monmap: 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:50.593 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 librados: starting msgr at 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 librados: starting objecter 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 Processor -- start 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 -- start start 2024-09-15T20:44:50.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 librados: setting wanted keys 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 librados: calling monclient init 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient: init 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient(hunting): picked mon.noname-a con 0x7f699c06e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient(hunting): picked mon.noname-c con 0x7f699c09f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient(hunting): picked mon.noname-b con 0x7f699c134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699c13dbb0 con 0x7f699c06e960 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699c11cea0 con 0x7f699c134bb0 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f699c0a43b0 con 0x7f699c09f5a0 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient(hunting): authenticate will time out at 1951.464792s 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a2101640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f699c06e960 0x7f699c1332f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57772/0 (socket says 172.21.15.78:57772) 2024-09-15T20:44:50.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a2101640 1 -- 172.21.15.78:0/2244929268 learned_addr learned my addr 172.21.15.78:0/2244929268 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a1900640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f699c09f5a0 0x7f699c1d0030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56478/0 (socket says 172.21.15.78:56478) 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.?.objecter ms_handle_connect 0x7f699c09f5a0 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.?.objecter ms_handle_connect 0x7f699c06e960 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2973741396 0 0) 0x7f699c13dbb0 con 0x7f699c06e960 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient(hunting): my global_id is 114324 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6980004ec0 con 0x7f699c06e960 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.?.objecter ms_handle_connect 0x7f699c134bb0 2024-09-15T20:44:50.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3177463961 0 0) 0x7f6980004ec0 con 0x7f699c06e960 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f699c13dbb0 con 0x7f699c06e960 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f698c001c10 con 0x7f699c06e960 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient(hunting): dump: 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.597 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3696700320 0 0) 0x7f699c13dbb0 con 0x7f699c06e960 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 >> v1:172.21.15.78:6790/0 conn(0x7f699c09f5a0 legacy=0x7f699c1d0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 >> v1:172.21.15.146:6789/0 conn(0x7f699c134bb0 legacy=0x7f699c1d3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 monclient: found mon.a 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f699c073a70 con 0x7f699c06e960 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient: _finish_auth 0 2024-09-15T20:44:50.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 5 monclient: authenticate success, global_id 114324 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient: _renew_subs 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f699c11cbc0 con 0x7f699c06e960 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 client.114324.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient: _renew_subs 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f699c13c5e0 con 0x7f699c06e960 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 librados: init done 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f698c001820 con 0x7f699c06e960 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f698c004e20 con 0x7f699c06e960 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 monclient: dump: 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a08fe640 4 set_mon_vals no callback set 2024-09-15T20:44:50.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f69840065d0 con 0x7f6984002870 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.114324.objecter ms_handle_connect 0x7f6984002870 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.114324.objecter resend_mon_ops 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f698c004fc0 con 0x7f699c06e960 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7f698c04ba30 con 0x7f699c06e960 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.114324.objecter ms_dispatch 0x7f699c000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 3 client.114324.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 3 client.114324.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 20 client.114324.objecter dump_active .. 0 homeless 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f69a2902640 10 monclient: get_auth_request con 0x7f698003dfc0 auth_method 0 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.578+0000 7f6992ffd640 10 client.114324.objecter ms_handle_connect 0x7f698003dfc0 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3062862104 0 0) 0x7f69840065d0 con 0x7f6984002870 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient(hunting): my global_id is 114327 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f699c13dbb0 con 0x7f6984002870 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 565663555 0 0) 0x7f699c13dbb0 con 0x7f6984002870 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f69840065d0 con 0x7f6984002870 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6998001750 con 0x7f6984002870 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient: dump: 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.602 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2336024974 0 0) 0x7f69840065d0 con 0x7f6984002870 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f698400a170 con 0x7f6984002870 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f6998001a90 con 0x7f6984002870 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 >> v1:172.21.15.78:6789/0 conn(0x7f6984002870 legacy=0x7f6984004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.582+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 >> v1:172.21.15.78:6789/0 conn(0x7f6984002870 legacy=0x7f6984004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:50.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f69a4365640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:44:50.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6984009e60 con 0x7f6984009120 2024-09-15T20:44:50.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 client.114324.objecter ms_handle_connect 0x7f6984009120 2024-09-15T20:44:50.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 client.114324.objecter resend_mon_ops 2024-09-15T20:44:50.605 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3025590486 0 0) 0x7f6984009e60 con 0x7f6984009120 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 monclient(hunting): my global_id is 114330 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f69840065d0 con 0x7f6984009120 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2813584045 0 0) 0x7f69840065d0 con 0x7f6984009120 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6984009e60 con 0x7f6984009120 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f69980037c0 con 0x7f6984009120 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 10 monclient: dump: 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.606 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1891641004 0 0) 0x7f6984009e60 con 0x7f6984009120 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.586+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f698400c400 con 0x7f6984009120 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4307 (unknown 0 0 3840973000) 0x7f69980057d0 con 0x7f6984009120 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f6992ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f6992ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 >> v1:172.21.15.78:6789/0 conn(0x7f6984009120 legacy=0x7f69840095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.607 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f6992ffd640 1 -- 172.21.15.78:0/2244929268 >> v1:172.21.15.78:6789/0 conn(0x7f6984009120 legacy=0x7f69840095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 54, 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 216429, 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:44:50.608 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.609 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.610 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:44:50.611 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.612 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:44:50.613 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:44:50.614 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.615 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.616 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.617 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:44:50.618 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.619 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.620 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.620 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:44:50.620 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:44:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 10 librados: watch_flush enter 2024-09-15T20:44:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 10 librados: watch_flush exit 2024-09-15T20:44:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 20 client.114324.objecter shutdown clearing up homeless session... 2024-09-15T20:44:50.621 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 10 client.114324.objecter successfully canceled tick 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 >> v1:172.21.15.146:6812/435598642 conn(0x7f698003dfc0 legacy=0x7f6980040480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 10 monclient: shutdown 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 >> v1:172.21.15.78:6789/0 conn(0x7f699c06e960 legacy=0x7f699c1332f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a2902640 1 -- 172.21.15.78:0/2244929268 reap_dead start 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a2902640 1 -- 172.21.15.78:0/2244929268 reap_dead start 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 shutdown_connections 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 >> 172.21.15.78:0/2244929268 conn(0x7f699c0a63c0 msgr2=0x7f699c0a1600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 shutdown_connections 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 1 -- 172.21.15.78:0/2244929268 wait complete. 2024-09-15T20:44:50.622 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.590+0000 7f69a4365640 1 librados: shutdown 2024-09-15T20:44:50.623 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:44:50.623 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:44:50.623 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:44:50.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient: get_monmap_and_config 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient: build_initial_monmap 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient: monmap: 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:50.741579+0000 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:50.741579+0000 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:50.740 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:50.741 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 1 Processor -- start 2024-09-15T20:44:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 1 -- start start 2024-09-15T20:44:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient: init 2024-09-15T20:44:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:44:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient(hunting): picked mon.noname-b con 0x7f67900a9160 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:50.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient(hunting): picked mon.noname-c con 0x7f67900a4e70 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient(hunting): picked mon.noname-a con 0x7f67900a9c50 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67900a5380 con 0x7f67900a9c50 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67900a63b0 con 0x7f67900a4e70 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67900a20a0 con 0x7f67900a9160 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient(hunting): authenticate will time out at 1951.626896s 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6795722640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f67900a9160 0x7f67900a9540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56486/0 (socket says 172.21.15.78:56486) 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6795722640 1 -- 172.21.15.78:0/3504404820 learned_addr learned my addr 172.21.15.78:0/3504404820 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 991439264 0 0) 0x7f67900a20a0 con 0x7f67900a9160 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient(hunting): my global_id is 124211 2024-09-15T20:44:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f677c003530 con 0x7f67900a9160 2024-09-15T20:44:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1894522077 0 0) 0x7f67900a5380 con 0x7f67900a9c50 2024-09-15T20:44:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient(hunting): my global_id is 114333 2024-09-15T20:44:50.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67900a20a0 con 0x7f67900a9c50 2024-09-15T20:44:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2531843660 0 0) 0x7f677c003530 con 0x7f67900a9160 2024-09-15T20:44:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 >> v1:172.21.15.146:6789/0 conn(0x7f67900a4e70 legacy=0x7f679013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 >> v1:172.21.15.78:6789/0 conn(0x7f67900a9c50 legacy=0x7f6790140600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: _finish_hunting 0 2024-09-15T20:44:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 monclient: found mon.noname-b 2024-09-15T20:44:50.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67900a7aa0 con 0x7f67900a9160 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: _finish_auth 0 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: _check_auth_tickets 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f678c0016a0 con 0x7f67900a9160 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 5 monclient: authenticate success, global_id 124211 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:44:50.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: dump: 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f678c0018c0 con 0x7f67900a9160 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 1 -- 172.21.15.78:0/3504404820 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f678c001d10 con 0x7f67900a9160 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6797986640 4 set_mon_vals no callback set 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.738+0000 7f6787fff640 10 monclient: dump: 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.749 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: shutdown 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- 172.21.15.78:0/3504404820 >> v1:172.21.15.78:6790/0 conn(0x7f67900a9160 legacy=0x7f67900a9540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- 172.21.15.78:0/3504404820 shutdown_connections 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- 172.21.15.78:0/3504404820 >> 172.21.15.78:0/3504404820 conn(0x7f67900a6d50 msgr2=0x7f67900a7130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- 172.21.15.78:0/3504404820 shutdown_connections 2024-09-15T20:44:50.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- 172.21.15.78:0/3504404820 wait complete. 2024-09-15T20:44:50.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: build_initial_monmap 2024-09-15T20:44:50.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: monmap: 2024-09-15T20:44:50.757 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:50.757 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 librados: starting msgr at 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 librados: starting objecter 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 Processor -- start 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- start start 2024-09-15T20:44:50.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 librados: setting wanted keys 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 librados: calling monclient init 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: init 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient(hunting): picked mon.noname-a con 0x7f67900a4e70 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient(hunting): picked mon.noname-c con 0x7f67900a9160 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient(hunting): picked mon.noname-b con 0x7f67900a9c50 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67900a1440 con 0x7f67900a4e70 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f679013feb0 con 0x7f67900a9c50 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f679013d600 con 0x7f67900a9160 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient(hunting): authenticate will time out at 1951.629840s 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6795722640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f67900a4e70 0x7f679013c7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57804/0 (socket says 172.21.15.78:57804) 2024-09-15T20:44:50.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6795722640 1 -- 172.21.15.78:0/1827747941 learned_addr learned my addr 172.21.15.78:0/1827747941 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 client.?.objecter ms_handle_connect 0x7f67900a4e70 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 client.?.objecter ms_handle_connect 0x7f67900a9160 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3396434086 0 0) 0x7f67900a7aa0 con 0x7f67900a4e70 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): my global_id is 114336 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6774004ec0 con 0x7f67900a4e70 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2220324965 0 0) 0x7f679013d600 con 0x7f67900a9160 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): my global_id is 124214 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67900a7aa0 con 0x7f67900a9160 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 client.?.objecter ms_handle_connect 0x7f67900a9c50 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 888478233 0 0) 0x7f6774004ec0 con 0x7f67900a4e70 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f679013d600 con 0x7f67900a4e70 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f678c001c10 con 0x7f67900a4e70 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): dump: 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.761 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 474792572 0 0) 0x7f67900a7aa0 con 0x7f67900a9160 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6774004ec0 con 0x7f67900a9160 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6780001880 con 0x7f67900a9160 2024-09-15T20:44:50.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient(hunting): dump: 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.763 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 962835090 0 0) 0x7f679013d600 con 0x7f67900a4e70 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 >> v1:172.21.15.78:6790/0 conn(0x7f67900a9160 legacy=0x7f679013cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 >> v1:172.21.15.146:6789/0 conn(0x7f67900a9c50 legacy=0x7f67901d4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 monclient: found mon.a 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6790141050 con 0x7f67900a4e70 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient: _finish_auth 0 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:50.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 5 monclient: authenticate success, global_id 114336 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: _renew_subs 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f67900a2a20 con 0x7f67900a4e70 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 client.114336.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: _renew_subs 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f679013fbd0 con 0x7f67900a4e70 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 librados: init done 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f678c001820 con 0x7f67900a4e70 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:50.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f678c004e20 con 0x7f67900a4e70 2024-09-15T20:44:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 10 monclient: dump: 2024-09-15T20:44:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.766 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6786ffd640 4 set_mon_vals no callback set 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f678c010e90 con 0x7f67900a4e70 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.742+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f67700065d0 con 0x7f6770002870 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7f678c04ba60 con 0x7f67900a4e70 2024-09-15T20:44:50.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 client.114336.objecter ms_dispatch 0x7f6790000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 3 client.114336.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 3 client.114336.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 20 client.114336.objecter dump_active .. 0 homeless 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 client.114336.objecter ms_handle_connect 0x7f6770002870 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 client.114336.objecter resend_mon_ops 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 20816923 0 0) 0x7f67700065d0 con 0x7f6770002870 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient(hunting): my global_id is 124217 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f679013d600 con 0x7f6770002870 2024-09-15T20:44:50.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6795f23640 10 monclient: get_auth_request con 0x7f677403e090 auth_method 0 2024-09-15T20:44:50.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 764385839 0 0) 0x7f679013d600 con 0x7f6770002870 2024-09-15T20:44:50.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f67700065d0 con 0x7f6770002870 2024-09-15T20:44:50.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f67800037a0 con 0x7f6770002870 2024-09-15T20:44:50.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:50.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient: dump: 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2936823614 0 0) 0x7f67700065d0 con 0x7f6770002870 2024-09-15T20:44:50.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f677000a170 con 0x7f6770002870 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 client.114336.objecter ms_handle_connect 0x7f677403e090 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f67800037c0 con 0x7f6770002870 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 >> v1:172.21.15.78:6790/0 conn(0x7f6770002870 legacy=0x7f6770004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.746+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 >> v1:172.21.15.78:6790/0 conn(0x7f6770002870 legacy=0x7f6770004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6797986640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f6770009e60 con 0x7f6770009120 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 client.114336.objecter ms_handle_connect 0x7f6770009120 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 client.114336.objecter resend_mon_ops 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1887328981 0 0) 0x7f6770009e60 con 0x7f6770009120 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient(hunting): my global_id is 124220 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67700065d0 con 0x7f6770009120 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 482925336 0 0) 0x7f67700065d0 con 0x7f6770009120 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6770009e60 con 0x7f6770009120 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6780003dd0 con 0x7f6770009120 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient: dump: 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.772 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1876503857 0 0) 0x7f6770009e60 con 0x7f6770009120 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f677000c3b0 con 0x7f6770009120 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 2367657651) 0x7f6780005360 con 0x7f6770009120 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 >> v1:172.21.15.78:6790/0 conn(0x7f6770009120 legacy=0x7f67700095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6785ffb640 1 -- 172.21.15.78:0/1827747941 >> v1:172.21.15.78:6790/0 conn(0x7f6770009120 legacy=0x7f67700095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:50.774 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:44:50.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:44:50.774 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:44:50.774 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:44:50.774 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 54, 2024-09-15T20:44:50.774 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 108678, 2024-09-15T20:44:50.774 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:44:50.774 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.775 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:44:50.776 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:44:50.778 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.779 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:44:50.780 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.781 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.782 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:44:50.783 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.784 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:44:50.785 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.786 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.786 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.786 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:44:50.786 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.750+0000 7f6797986640 10 librados: watch_flush enter 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 10 librados: watch_flush exit 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 20 client.114336.objecter shutdown clearing up homeless session... 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 10 client.114336.objecter successfully canceled tick 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 >> v1:172.21.15.146:6812/435598642 conn(0x7f677403e090 legacy=0x7f6774040550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 10 monclient: shutdown 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 >> v1:172.21.15.78:6789/0 conn(0x7f67900a4e70 legacy=0x7f679013c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6795f23640 1 -- 172.21.15.78:0/1827747941 reap_dead start 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6795f23640 1 -- 172.21.15.78:0/1827747941 reap_dead start 2024-09-15T20:44:50.788 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 shutdown_connections 2024-09-15T20:44:50.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 >> 172.21.15.78:0/1827747941 conn(0x7f67900a6d50 msgr2=0x7f67900a0350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:50.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 shutdown_connections 2024-09-15T20:44:50.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 1 -- 172.21.15.78:0/1827747941 wait complete. 2024-09-15T20:44:50.789 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.754+0000 7f6797986640 1 librados: shutdown 2024-09-15T20:44:50.789 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:44:50.789 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:44:50.789 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:44:50.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient: get_monmap_and_config 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient: build_initial_monmap 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient: monmap: 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:50.897636+0000 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:50.897636+0000 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:50.896 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:50.897 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 Processor -- start 2024-09-15T20:44:50.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- start start 2024-09-15T20:44:50.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient: init 2024-09-15T20:44:50.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:50.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:44:50.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient(hunting): picked mon.noname-a con 0x7fc360134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:50.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient(hunting): picked mon.noname-c con 0x7fc36009f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:50.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient(hunting): picked mon.noname-b con 0x7fc36006e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3600a0470 con 0x7fc360134bb0 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc36009feb0 con 0x7fc36009f5a0 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc36009e4a0 con 0x7fc36006e960 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient(hunting): authenticate will time out at 1951.782283s 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35ed9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc36006e960 0x7fc36013d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56534/0 (socket says 172.21.15.78:56534) 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35ed9d640 1 -- 172.21.15.78:0/1511764447 learned_addr learned my addr 172.21.15.78:0/1511764447 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35e59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc360134bb0 0x7fc360134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57814/0 (socket says 172.21.15.78:57814) 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2236835566 0 0) 0x7fc3580013d0 con 0x7fc360134bb0 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient(hunting): my global_id is 114339 2024-09-15T20:44:50.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc348003530 con 0x7fc360134bb0 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254138307 0 0) 0x7fc36009e4a0 con 0x7fc36006e960 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient(hunting): my global_id is 124223 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3580013d0 con 0x7fc36006e960 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4242617578 0 0) 0x7fc348003530 con 0x7fc360134bb0 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 >> v1:172.21.15.78:6790/0 conn(0x7fc36006e960 legacy=0x7fc36013d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 >> v1:172.21.15.146:6789/0 conn(0x7fc36009f5a0 legacy=0x7fc3601399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: _finish_hunting 0 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 monclient: found mon.noname-a 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:44:50.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3600a0ff0 con 0x7fc360134bb0 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: _finish_auth 0 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: _check_auth_tickets 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc358001b20 con 0x7fc360134bb0 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: dump: 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.908 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 5 monclient: authenticate success, global_id 114339 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc358001d40 con 0x7fc360134bb0 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 1 -- 172.21.15.78:0/1511764447 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc358004b70 con 0x7fc360134bb0 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 4 set_mon_vals no callback set 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:50.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc35d59a640 10 monclient: dump: 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 10 monclient: shutdown 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:50.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- 172.21.15.78:0/1511764447 >> v1:172.21.15.78:6789/0 conn(0x7fc360134bb0 legacy=0x7fc360134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- 172.21.15.78:0/1511764447 shutdown_connections 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- 172.21.15.78:0/1511764447 >> 172.21.15.78:0/1511764447 conn(0x7fc3600a63c0 msgr2=0x7fc3600a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- 172.21.15.78:0/1511764447 shutdown_connections 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.894+0000 7fc36489e640 1 -- 172.21.15.78:0/1511764447 wait complete. 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: build_initial_monmap 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: monmap: 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:50.911 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 librados: starting msgr at 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 librados: starting objecter 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 Processor -- start 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 -- start start 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 librados: setting wanted keys 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 librados: calling monclient init 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: init 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:50.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient(hunting): picked mon.noname-c con 0x7fc36006e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient(hunting): picked mon.noname-b con 0x7fc36009f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient(hunting): picked mon.noname-a con 0x7fc360134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc360122370 con 0x7fc360134bb0 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc36009d180 con 0x7fc36009f5a0 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3601cd430 con 0x7fc36006e960 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient(hunting): authenticate will time out at 1951.785262s 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc35e59c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc36006e960 0x7fc360132f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56542/0 (socket says 172.21.15.78:56542) 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc35ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc360134bb0 0x7fc3601d52e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57820/0 (socket says 172.21.15.78:57820) 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc35e59c640 1 -- 172.21.15.78:0/3847464489 learned_addr learned my addr 172.21.15.78:0/3847464489 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.?.objecter ms_handle_connect 0x7fc360134bb0 2024-09-15T20:44:50.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.?.objecter ms_handle_connect 0x7fc36006e960 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 110643392 0 0) 0x7fc3601cd430 con 0x7fc36006e960 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): my global_id is 124226 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc340004ec0 con 0x7fc36006e960 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1645811413 0 0) 0x7fc360122370 con 0x7fc360134bb0 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): my global_id is 114342 2024-09-15T20:44:50.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3601cd430 con 0x7fc360134bb0 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.?.objecter ms_handle_connect 0x7fc36009f5a0 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3418837354 0 0) 0x7fc340004ec0 con 0x7fc36006e960 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc360122370 con 0x7fc36006e960 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc3580014c0 con 0x7fc36006e960 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): dump: 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.915 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4051255331 0 0) 0x7fc3601cd430 con 0x7fc360134bb0 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc340004ec0 con 0x7fc360134bb0 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc3500012e0 con 0x7fc360134bb0 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): dump: 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.916 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1503468783 0 0) 0x7fc36009d180 con 0x7fc36009f5a0 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): my global_id is 134107 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3601cd430 con 0x7fc36009f5a0 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3516641058 0 0) 0x7fc360122370 con 0x7fc36006e960 2024-09-15T20:44:50.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 >> v1:172.21.15.146:6789/0 conn(0x7fc36009f5a0 legacy=0x7fc3601d1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 >> v1:172.21.15.78:6789/0 conn(0x7fc360134bb0 legacy=0x7fc3601d52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient: _finish_hunting 0 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 monclient: found mon.c 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc360123e30 con 0x7fc36006e960 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient: _finish_auth 0 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient: _check_auth_tickets 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 5 monclient: authenticate success, global_id 124226 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: _renew_subs 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc360133610 con 0x7fc36006e960 2024-09-15T20:44:50.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 client.124226.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: _renew_subs 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc3601230a0 con 0x7fc36006e960 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 librados: init done 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc358001ca0 con 0x7fc36006e960 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc358004fc0 con 0x7fc36006e960 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc35cd99640 4 set_mon_vals no callback set 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient: dump: 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.919 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc3440065d0 con 0x7fc344002870 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fc3580051e0 con 0x7fc36006e960 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.124226.objecter ms_handle_connect 0x7fc344002870 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.124226.objecter resend_mon_ops 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7fc35804bc10 con 0x7fc36006e960 2024-09-15T20:44:50.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.124226.objecter ms_dispatch 0x7fc360000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 3 client.124226.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 3 client.124226.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 20 client.124226.objecter dump_active .. 0 homeless 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2860671262 0 0) 0x7fc3440065d0 con 0x7fc344002870 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 monclient(hunting): my global_id is 134110 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc360122370 con 0x7fc344002870 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc35ed9d640 10 monclient: get_auth_request con 0x7fc34003e260 auth_method 0 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.898+0000 7fc34effd640 10 client.124226.objecter ms_handle_connect 0x7fc34003e260 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2443968446 0 0) 0x7fc360122370 con 0x7fc344002870 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc3440065d0 con 0x7fc344002870 2024-09-15T20:44:50.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc3540017e0 con 0x7fc344002870 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 10 monclient: dump: 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1230587378 0 0) 0x7fc3440065d0 con 0x7fc344002870 2024-09-15T20:44:50.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc34400a170 con 0x7fc344002870 2024-09-15T20:44:50.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc354001aa0 con 0x7fc344002870 2024-09-15T20:44:50.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:50.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:50.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 >> v1:172.21.15.146:6789/0 conn(0x7fc344002870 legacy=0x7fc344004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.902+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 >> v1:172.21.15.146:6789/0 conn(0x7fc344002870 legacy=0x7fc344004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.906+0000 7fc36489e640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:44:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.906+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc344009e80 con 0x7fc344009140 2024-09-15T20:44:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.906+0000 7fc34effd640 10 client.124226.objecter ms_handle_connect 0x7fc344009140 2024-09-15T20:44:50.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.906+0000 7fc34effd640 10 client.124226.objecter resend_mon_ops 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 380347728 0 0) 0x7fc344009e80 con 0x7fc344009140 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient(hunting): my global_id is 134113 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3440065d0 con 0x7fc344009140 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2310367453 0 0) 0x7fc3440065d0 con 0x7fc344009140 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc344009e80 con 0x7fc344009140 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc3540037a0 con 0x7fc344009140 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient: dump: 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:50.925 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3093588582 0 0) 0x7fc344009e80 con 0x7fc344009140 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc34400c420 con 0x7fc344009140 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 1567460962) 0x7fc354003730 con 0x7fc344009140 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:44:50.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 >> v1:172.21.15.146:6789/0 conn(0x7fc344009140 legacy=0x7fc3440095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc34effd640 1 -- 172.21.15.78:0/3847464489 >> v1:172.21.15.146:6789/0 conn(0x7fc344009140 legacy=0x7fc3440095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 54, 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 779, 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:44:50.927 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.928 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.929 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:44:50.930 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:50.931 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.932 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.933 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.934 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.935 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.936 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:50.937 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:50.938 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:50.938 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:50.938 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:44:50.938 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:44:50.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 10 librados: watch_flush enter 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 10 librados: watch_flush exit 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 20 client.124226.objecter shutdown clearing up homeless session... 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 10 client.124226.objecter successfully canceled tick 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 >> v1:172.21.15.146:6812/435598642 conn(0x7fc34003e260 legacy=0x7fc340040720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 10 monclient: shutdown 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 >> v1:172.21.15.78:6790/0 conn(0x7fc36006e960 legacy=0x7fc360132f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc35ed9d640 1 -- 172.21.15.78:0/3847464489 reap_dead start 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc35ed9d640 1 -- 172.21.15.78:0/3847464489 reap_dead start 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 shutdown_connections 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 >> 172.21.15.78:0/3847464489 conn(0x7fc3600a63c0 msgr2=0x7fc3600a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:50.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 shutdown_connections 2024-09-15T20:44:50.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 1 -- 172.21.15.78:0/3847464489 wait complete. 2024-09-15T20:44:50.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:50.910+0000 7fc36489e640 1 librados: shutdown 2024-09-15T20:44:50.941 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:44:50.941 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:44:50.941 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:44:51.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.046+0000 7f67248ba640 10 monclient: get_monmap_and_config 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.046+0000 7f67248ba640 10 monclient: build_initial_monmap 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.046+0000 7f67248ba640 10 monclient: monmap: 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:51.051771+0000 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:51.051771+0000 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:51.050 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 Processor -- start 2024-09-15T20:44:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- start start 2024-09-15T20:44:51.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: init 2024-09-15T20:44:51.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:51.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6720137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6720074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient(hunting): picked mon.noname-b con 0x7f672013e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6720138850 con 0x7f6720137d60 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67200a5da0 con 0x7f6720074500 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6720073560 con 0x7f672013e370 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient(hunting): authenticate will time out at 1951.936995s 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671e59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6720137d60 0x7f6720138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57826/0 (socket says 172.21.15.78:57826) 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671e59c640 1 -- 172.21.15.78:0/1040007143 learned_addr learned my addr 172.21.15.78:0/1040007143 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671ed9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f672013e370 0x7f6720140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56544/0 (socket says 172.21.15.78:56544) 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 654700945 0 0) 0x7f6720138850 con 0x7f6720137d60 2024-09-15T20:44:51.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient(hunting): _negotiate 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient(hunting): my global_id is 114345 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6708003530 con 0x7f6720137d60 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1772569915 0 0) 0x7f6720073560 con 0x7f672013e370 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient(hunting): _negotiate 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient(hunting): my global_id is 124229 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6720138850 con 0x7f672013e370 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1614302591 0 0) 0x7f6708003530 con 0x7f6720137d60 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 >> v1:172.21.15.78:6790/0 conn(0x7f672013e370 legacy=0x7f6720140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 >> v1:172.21.15.146:6789/0 conn(0x7f6720074500 legacy=0x7f672013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:51.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: _finish_hunting 0 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 monclient: found mon.noname-a 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6720073840 con 0x7f6720137d60 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: _finish_auth 0 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: _check_auth_tickets 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f67100016a0 con 0x7f6720137d60 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 5 monclient: authenticate success, global_id 114345 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:51.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: dump: 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6710001840 con 0x7f6720137d60 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:51.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 1 -- 172.21.15.78:0/1040007143 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6710001c50 con 0x7f6720137d60 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 4 set_mon_vals no callback set 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f671d59a640 10 monclient: dump: 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:51.062 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: shutdown 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- 172.21.15.78:0/1040007143 >> v1:172.21.15.78:6789/0 conn(0x7f6720137d60 legacy=0x7f6720138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- 172.21.15.78:0/1040007143 shutdown_connections 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- 172.21.15.78:0/1040007143 >> 172.21.15.78:0/1040007143 conn(0x7f67200aa650 msgr2=0x7f67200aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- 172.21.15.78:0/1040007143 shutdown_connections 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- 172.21.15.78:0/1040007143 wait complete. 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: build_initial_monmap 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: monmap: 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:51.063 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 librados: starting msgr at 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 librados: starting objecter 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 Processor -- start 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 -- start start 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 librados: setting wanted keys 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 1 librados: calling monclient init 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: init 2024-09-15T20:44:51.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient(hunting): picked mon.noname-c con 0x7f6720074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient(hunting): picked mon.noname-a con 0x7f6720137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.050+0000 7f67248ba640 10 monclient(hunting): picked mon.noname-b con 0x7f672013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f67201327e0 con 0x7f6720137d60 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6720132ac0 con 0x7f672013e370 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f672009f380 con 0x7f6720074500 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 monclient(hunting): authenticate will time out at 1951.940046s 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f671e59c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f6720074500 0x7f672009e560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56560/0 (socket says 172.21.15.78:56560) 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f671e59c640 1 -- 172.21.15.78:0/2324465821 learned_addr learned my addr 172.21.15.78:0/2324465821 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 client.?.objecter ms_handle_connect 0x7f6720137d60 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:51.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 client.?.objecter ms_handle_connect 0x7f6720074500 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3666570928 0 0) 0x7f67201327e0 con 0x7f6720137d60 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): my global_id is 114348 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6700004ec0 con 0x7f6720137d60 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2933456000 0 0) 0x7f672009f380 con 0x7f6720074500 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): my global_id is 124232 2024-09-15T20:44:51.066 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f67201327e0 con 0x7f6720074500 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 client.?.objecter ms_handle_connect 0x7f672013e370 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3148481320 0 0) 0x7f6700004ec0 con 0x7f6720137d60 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f672009f380 con 0x7f6720137d60 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6714001940 con 0x7f6720137d60 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): dump: 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:51.067 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2138009912 0 0) 0x7f67201327e0 con 0x7f6720074500 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6700004ec0 con 0x7f6720074500 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6710001b90 con 0x7f6720074500 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient(hunting): dump: 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:51.068 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3165469309 0 0) 0x7f672009f380 con 0x7f6720137d60 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 >> v1:172.21.15.78:6790/0 conn(0x7f6720074500 legacy=0x7f672009e560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 >> v1:172.21.15.146:6789/0 conn(0x7f672013e370 legacy=0x7f67201d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient: _finish_hunting 0 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 monclient: found mon.a 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:51.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6720141740 con 0x7f6720137d60 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient: _finish_auth 0 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient: _check_auth_tickets 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 5 monclient: authenticate success, global_id 114348 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 monclient: _renew_subs 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f672009fee0 con 0x7f6720137d60 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 client.114348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 monclient: _renew_subs 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f67200a05c0 con 0x7f6720137d60 2024-09-15T20:44:51.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 1 librados: init done 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6714003600 con 0x7f6720137d60 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f67140039d0 con 0x7f6720137d60 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 monclient: dump: 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:51.071 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f671cd99640 4 set_mon_vals no callback set 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6704005090 con 0x7f6720137d60 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f6714003b70 con 0x7f6720137d60 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7f671404a780 con 0x7f6720137d60 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 client.114348.objecter ms_dispatch 0x7f6720000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 3 client.114348.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 3 client.114348.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 20 client.114348.objecter dump_active .. 0 homeless 2024-09-15T20:44:51.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f671e59c640 10 monclient: get_auth_request con 0x7f670003e0d0 auth_method 0 2024-09-15T20:44:51.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.054+0000 7f670effd640 10 client.114348.objecter ms_handle_connect 0x7f670003e0d0 2024-09-15T20:44:51.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.058+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f6714015940 con 0x7f6720137d60 2024-09-15T20:44:51.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.058+0000 7f670effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:51.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.058+0000 7f670effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:51.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.282+0000 7f67248ba640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:44:51.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.282+0000 7f67248ba640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:44:51.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.282+0000 7f67248ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:51.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.282+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f6704005c50 con 0x7f6720137d60 2024-09-15T20:44:51.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f670effd640 1 -- 172.21.15.78:0/2324465821 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f6714015240 con 0x7f6720137d60 2024-09-15T20:44:51.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f670effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:44:51.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f670effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:51.288 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.288 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=2636729021,config=2586702516,health=1126638305,kv=199072906,logm=3728699097}) 2024-09-15T20:44:51.288 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=20,config=2,health=12,kv=12,logm=54} crc {auth=3142650668,config=2586702516,health=1126638305,kv=2846337045,logm=76083353}) 2024-09-15T20:44:51.288 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.288 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=2636729021,config=2586702516,health=1126638305,kv=199072906,logm=3728699097}) 2024-09-15T20:44:51.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=21,config=2,health=12,kv=12,logm=53} crc {auth=3494266759,config=2586702516,health=2925248371,kv=852237645,logm=4225671421}) 2024-09-15T20:44:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 10 librados: watch_flush enter 2024-09-15T20:44:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 10 librados: watch_flush exit 2024-09-15T20:44:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 20 client.114348.objecter shutdown clearing up homeless session... 2024-09-15T20:44:51.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 10 client.114348.objecter successfully canceled tick 2024-09-15T20:44:51.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 >> v1:172.21.15.146:6812/435598642 conn(0x7f670003e0d0 legacy=0x7f6700040590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:51.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 10 monclient: shutdown 2024-09-15T20:44:51.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:51.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 >> v1:172.21.15.78:6789/0 conn(0x7f6720137d60 legacy=0x7f672009ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:51.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 shutdown_connections 2024-09-15T20:44:51.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 >> 172.21.15.78:0/2324465821 conn(0x7f67200aa650 msgr2=0x7f6720130b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:51.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 shutdown_connections 2024-09-15T20:44:51.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 1 -- 172.21.15.78:0/2324465821 wait complete. 2024-09-15T20:44:51.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:51.286+0000 7f67248ba640 1 librados: shutdown 2024-09-15T20:44:51.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388523108}) 2024-09-15T20:44:51.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2022078695}) 2024-09-15T20:44:51.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2388523108}) 2024-09-15T20:44:51.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.286+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2293890056}) 2024-09-15T20:44:51.292 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210866674}) 2024-09-15T20:44:51.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=295895352}) 2024-09-15T20:44:51.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2210866674}) 2024-09-15T20:44:51.293 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1291435933}) 2024-09-15T20:44:51.294 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.294 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675426673}) 2024-09-15T20:44:51.294 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=450699981}) 2024-09-15T20:44:51.294 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.294 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3675426673}) 2024-09-15T20:44:51.295 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.290+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2369188108}) 2024-09-15T20:44:51.295 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.296 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025195282}) 2024-09-15T20:44:51.296 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=54088670}) 2024-09-15T20:44:51.296 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.296 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2025195282}) 2024-09-15T20:44:51.296 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4239657823}) 2024-09-15T20:44:51.297 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.298 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622446780}) 2024-09-15T20:44:51.298 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3732541666}) 2024-09-15T20:44:51.298 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.298 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3622446780}) 2024-09-15T20:44:51.298 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.294+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4038616496}) 2024-09-15T20:44:51.299 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.299 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692949265}) 2024-09-15T20:44:51.300 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=640595720}) 2024-09-15T20:44:51.300 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.300 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3692949265}) 2024-09-15T20:44:51.300 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2783364304}) 2024-09-15T20:44:51.301 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.301 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119720865}) 2024-09-15T20:44:51.301 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=651208801}) 2024-09-15T20:44:51.301 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.301 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2119720865}) 2024-09-15T20:44:51.301 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.298+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258566497}) 2024-09-15T20:44:51.303 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.303 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376000998}) 2024-09-15T20:44:51.303 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4097839431}) 2024-09-15T20:44:51.303 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.303 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=376000998}) 2024-09-15T20:44:51.304 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=438646997}) 2024-09-15T20:44:51.305 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.305 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2658973836}) 2024-09-15T20:44:51.305 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2589324386}) 2024-09-15T20:44:51.305 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.305 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2658973836}) 2024-09-15T20:44:51.305 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=453685689}) 2024-09-15T20:44:51.306 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:44:51.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188743391}) 2024-09-15T20:44:51.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2281970555}) 2024-09-15T20:44:51.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1188743391}) 2024-09-15T20:44:51.307 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.302+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1371369135}) 2024-09-15T20:44:51.308 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.308 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371009660}) 2024-09-15T20:44:51.308 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2817536050}) 2024-09-15T20:44:51.308 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.309 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2371009660}) 2024-09-15T20:44:51.309 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2882454495}) 2024-09-15T20:44:51.310 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.310 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397405108}) 2024-09-15T20:44:51.311 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=473111937}) 2024-09-15T20:44:51.311 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.311 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2397405108}) 2024-09-15T20:44:51.311 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.306+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1997812774}) 2024-09-15T20:44:51.312 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.313 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859634434}) 2024-09-15T20:44:51.313 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2891601130}) 2024-09-15T20:44:51.313 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.313 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=859634434}) 2024-09-15T20:44:51.313 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3771725173}) 2024-09-15T20:44:51.314 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.314 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013474819}) 2024-09-15T20:44:51.314 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2945326123}) 2024-09-15T20:44:51.315 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.315 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2013474819}) 2024-09-15T20:44:51.315 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.310+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1521587997}) 2024-09-15T20:44:51.316 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.316 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594255903}) 2024-09-15T20:44:51.316 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3904321640}) 2024-09-15T20:44:51.316 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.316 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3594255903}) 2024-09-15T20:44:51.316 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=910537448}) 2024-09-15T20:44:51.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518301296}) 2024-09-15T20:44:51.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1381053404}) 2024-09-15T20:44:51.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=518301296}) 2024-09-15T20:44:51.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.314+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1435942095}) 2024-09-15T20:44:51.321 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.321 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=404162316}) 2024-09-15T20:44:51.321 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=908245405}) 2024-09-15T20:44:51.321 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.321 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=404162316}) 2024-09-15T20:44:51.321 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1799620645}) 2024-09-15T20:44:51.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581850133}) 2024-09-15T20:44:51.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1943335350}) 2024-09-15T20:44:51.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.318+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2581850133}) 2024-09-15T20:44:51.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.322+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2152890538}) 2024-09-15T20:44:51.332 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.332 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002308910}) 2024-09-15T20:44:51.332 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2193894608}) 2024-09-15T20:44:51.332 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.333 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3002308910}) 2024-09-15T20:44:51.333 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1104262326}) 2024-09-15T20:44:51.334 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.334 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218281581}) 2024-09-15T20:44:51.334 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1850350483}) 2024-09-15T20:44:51.334 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.334 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3218281581}) 2024-09-15T20:44:51.334 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.330+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2918707999}) 2024-09-15T20:44:51.335 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177622493}) 2024-09-15T20:44:51.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=575290975}) 2024-09-15T20:44:51.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3177622493}) 2024-09-15T20:44:51.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2678449834}) 2024-09-15T20:44:51.337 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.337 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1129806659}) 2024-09-15T20:44:51.338 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2541294303}) 2024-09-15T20:44:51.338 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.338 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1129806659}) 2024-09-15T20:44:51.338 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.334+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1579908783}) 2024-09-15T20:44:51.339 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.339 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635730468}) 2024-09-15T20:44:51.339 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=759283582}) 2024-09-15T20:44:51.339 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.339 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2635730468}) 2024-09-15T20:44:51.339 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=442104064}) 2024-09-15T20:44:51.341 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.341 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733260822}) 2024-09-15T20:44:51.341 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=119491723}) 2024-09-15T20:44:51.341 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.341 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=733260822}) 2024-09-15T20:44:51.341 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3742462408}) 2024-09-15T20:44:51.343 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.343 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164154956}) 2024-09-15T20:44:51.343 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1231905373}) 2024-09-15T20:44:51.343 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.343 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1164154956}) 2024-09-15T20:44:51.343 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.338+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1739422784}) 2024-09-15T20:44:51.344 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.344 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1766359767}) 2024-09-15T20:44:51.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=94864001}) 2024-09-15T20:44:51.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1766359767}) 2024-09-15T20:44:51.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=58490295}) 2024-09-15T20:44:51.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3054708716}) 2024-09-15T20:44:51.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1690486207}) 2024-09-15T20:44:51.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3054708716}) 2024-09-15T20:44:51.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.342+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=135921571}) 2024-09-15T20:44:51.347 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.347 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075800734}) 2024-09-15T20:44:51.348 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=745016278}) 2024-09-15T20:44:51.348 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.348 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075800734}) 2024-09-15T20:44:51.348 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=938372028}) 2024-09-15T20:44:51.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750925646}) 2024-09-15T20:44:51.350 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71717412}) 2024-09-15T20:44:51.350 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.350 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2750925646}) 2024-09-15T20:44:51.350 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.346+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2543795081}) 2024-09-15T20:44:51.351 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3628074381}) 2024-09-15T20:44:51.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4191325345}) 2024-09-15T20:44:51.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3628074381}) 2024-09-15T20:44:51.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2912378716}) 2024-09-15T20:44:51.353 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.353 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034499861}) 2024-09-15T20:44:51.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3217728127}) 2024-09-15T20:44:51.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4034499861}) 2024-09-15T20:44:51.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.350+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3013727761}) 2024-09-15T20:44:51.355 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.355 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63424989}) 2024-09-15T20:44:51.355 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3298018841}) 2024-09-15T20:44:51.355 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.355 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=63424989}) 2024-09-15T20:44:51.355 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=476509231}) 2024-09-15T20:44:51.357 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.357 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385988638}) 2024-09-15T20:44:51.357 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2416274789}) 2024-09-15T20:44:51.357 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.357 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2385988638}) 2024-09-15T20:44:51.357 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=773935124}) 2024-09-15T20:44:51.359 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.359 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.354+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963404355}) 2024-09-15T20:44:51.359 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3283382420}) 2024-09-15T20:44:51.359 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.359 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2963404355}) 2024-09-15T20:44:51.359 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3002355398}) 2024-09-15T20:44:51.360 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.361 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802147641}) 2024-09-15T20:44:51.361 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3966389877}) 2024-09-15T20:44:51.361 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.361 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=802147641}) 2024-09-15T20:44:51.361 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3142404254}) 2024-09-15T20:44:51.362 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118361235}) 2024-09-15T20:44:51.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=688046512}) 2024-09-15T20:44:51.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3118361235}) 2024-09-15T20:44:51.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.358+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1160802618}) 2024-09-15T20:44:51.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332330713}) 2024-09-15T20:44:51.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2583317276}) 2024-09-15T20:44:51.365 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.365 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2332330713}) 2024-09-15T20:44:51.365 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3275715108}) 2024-09-15T20:44:51.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155619104}) 2024-09-15T20:44:51.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2293514939}) 2024-09-15T20:44:51.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=155619104}) 2024-09-15T20:44:51.367 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.362+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1636310085}) 2024-09-15T20:44:51.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294648714}) 2024-09-15T20:44:51.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3532817446}) 2024-09-15T20:44:51.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3294648714}) 2024-09-15T20:44:51.369 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3509577883}) 2024-09-15T20:44:51.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2913379911}) 2024-09-15T20:44:51.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1422855505}) 2024-09-15T20:44:51.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2913379911}) 2024-09-15T20:44:51.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.366+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1186977754}) 2024-09-15T20:44:51.372 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.372 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1614579492}) 2024-09-15T20:44:51.372 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1914227162}) 2024-09-15T20:44:51.372 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.372 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1614579492}) 2024-09-15T20:44:51.372 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1138210639}) 2024-09-15T20:44:51.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348887825}) 2024-09-15T20:44:51.374 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3056753910}) 2024-09-15T20:44:51.374 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.374 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3348887825}) 2024-09-15T20:44:51.374 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.370+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3274825128}) 2024-09-15T20:44:51.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.376 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436073387}) 2024-09-15T20:44:51.376 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=648497826}) 2024-09-15T20:44:51.376 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.376 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3436073387}) 2024-09-15T20:44:51.376 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3122331625}) 2024-09-15T20:44:51.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2868260311}) 2024-09-15T20:44:51.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3921880695}) 2024-09-15T20:44:51.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2868260311}) 2024-09-15T20:44:51.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.374+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3118863263}) 2024-09-15T20:44:51.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477391174}) 2024-09-15T20:44:51.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=190788719}) 2024-09-15T20:44:51.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477391174}) 2024-09-15T20:44:51.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=860329330}) 2024-09-15T20:44:51.381 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.381 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697349020}) 2024-09-15T20:44:51.381 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4188563122}) 2024-09-15T20:44:51.381 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697349020}) 2024-09-15T20:44:51.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3293224920}) 2024-09-15T20:44:51.383 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.383 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833195295}) 2024-09-15T20:44:51.383 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3843472779}) 2024-09-15T20:44:51.383 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.383 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2833195295}) 2024-09-15T20:44:51.383 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.378+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1576811040}) 2024-09-15T20:44:51.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221079790}) 2024-09-15T20:44:51.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1302814233}) 2024-09-15T20:44:51.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2221079790}) 2024-09-15T20:44:51.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2808709835}) 2024-09-15T20:44:51.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313787778}) 2024-09-15T20:44:51.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=110581157}) 2024-09-15T20:44:51.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=313787778}) 2024-09-15T20:44:51.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.382+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=116929062}) 2024-09-15T20:44:51.388 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.388 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93437330}) 2024-09-15T20:44:51.388 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2512001506}) 2024-09-15T20:44:51.388 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.388 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=93437330}) 2024-09-15T20:44:51.388 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1846932762}) 2024-09-15T20:44:51.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448321304}) 2024-09-15T20:44:51.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2283835570}) 2024-09-15T20:44:51.390 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.390 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=448321304}) 2024-09-15T20:44:51.390 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.386+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=694411730}) 2024-09-15T20:44:51.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727774344}) 2024-09-15T20:44:51.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3164647204}) 2024-09-15T20:44:51.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3727774344}) 2024-09-15T20:44:51.392 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3377615217}) 2024-09-15T20:44:51.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117831551}) 2024-09-15T20:44:51.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3445901814}) 2024-09-15T20:44:51.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=117831551}) 2024-09-15T20:44:51.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3524686913}) 2024-09-15T20:44:51.394 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.394 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020062367}) 2024-09-15T20:44:51.394 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2445293724}) 2024-09-15T20:44:51.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4020062367}) 2024-09-15T20:44:51.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.390+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3992934306}) 2024-09-15T20:44:51.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=24,mdsmap=2,mgr=7,mgr_command_descs=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=3543821542,mdsmap=3140514009,mgr=2467901132,mgr_command_descs=2551042219,mgrstat=463100327,mon_config_key=2432104054,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1681388020,osd_pg_creating=1110218226,osd_snap=3479108705}) 2024-09-15T20:44:51.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=24,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=2,nvmeofgw=3,osd_metadata=5,osd_snap=29} crc {logm=550918269,mdsmap=503983409,mgr=3781860113,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=643891099,mon_config_key=128973909,monmap=3140514009,nvmeofgw=598337686,osd_metadata=3213783154,osd_snap=3538633639}) 2024-09-15T20:44:51.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=24,mdsmap=2,mgr=7,mgr_command_descs=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=3543821542,mdsmap=3140514009,mgr=2467901132,mgr_command_descs=2551042219,mgrstat=463100327,mon_config_key=2432104054,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1681388020,osd_pg_creating=1110218226,osd_snap=3479108705}) 2024-09-15T20:44:51.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=22,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=8,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=4,osd_pg_creating=1,osd_snap=29} crc {logm=2081457860,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=1389705013,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=598337686,osd_metadata=3686268242,osd_pg_creating=1110218226,osd_snap=4176818418}) 2024-09-15T20:44:51.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.399 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=1183288867,osdmap=3972086826}) 2024-09-15T20:44:51.399 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=3,osdmap=97} crc {osd_snap=2411610781,osdmap=1674525088}) 2024-09-15T20:44:51.399 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.399 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=1183288867,osdmap=3972086826}) 2024-09-15T20:44:51.399 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.394+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=4,osdmap=96} crc {osd_snap=1183288867,osdmap=1067429316}) 2024-09-15T20:44:51.401 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.401 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=887869651}) 2024-09-15T20:44:51.401 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3347366761}) 2024-09-15T20:44:51.401 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.401 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=887869651}) 2024-09-15T20:44:51.401 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1044244346}) 2024-09-15T20:44:51.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=3492806358}) 2024-09-15T20:44:51.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=47} crc {osdmap=1265231323}) 2024-09-15T20:44:51.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:44:51.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=45} crc {osdmap=3492806358}) 2024-09-15T20:44:51.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:44:51.398+0000 7fe6455ff640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=46} crc {osdmap=2990552179}) 2024-09-15T20:44:52.306 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:44:52.307 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:44:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient: get_monmap_and_config 2024-09-15T20:44:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient: build_initial_monmap 2024-09-15T20:44:52.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient: monmap: 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:52.442303+0000 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:52.442303+0000 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:52.441 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 1 Processor -- start 2024-09-15T20:44:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 1 -- start start 2024-09-15T20:44:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient: init 2024-09-15T20:44:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:44:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient(hunting): picked mon.noname-a con 0x7f4d88135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient(hunting): picked mon.noname-c con 0x7f4d880a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:52.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient(hunting): picked mon.noname-b con 0x7f4d8813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d880745e0 con 0x7f4d88135b60 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d8806ecf0 con 0x7f4d880a24e0 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d8806efd0 con 0x7f4d8813c220 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient(hunting): authenticate will time out at 1953.326877s 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8e111640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f4d8813c220 0x7f4d8813e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56576/0 (socket says 172.21.15.78:56576) 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8e111640 1 -- 172.21.15.78:0/3655151104 learned_addr learned my addr 172.21.15.78:0/3655151104 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145267724 0 0) 0x7f4d8806efd0 con 0x7f4d8813c220 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient(hunting): my global_id is 124235 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d70003530 con 0x7f4d8813c220 2024-09-15T20:44:52.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2399797192 0 0) 0x7f4d880745e0 con 0x7f4d88135b60 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient(hunting): my global_id is 114351 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d8806efd0 con 0x7f4d88135b60 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3884681336 0 0) 0x7f4d8806efd0 con 0x7f4d88135b60 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 >> v1:172.21.15.78:6790/0 conn(0x7f4d8813c220 legacy=0x7f4d8813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 >> v1:172.21.15.146:6789/0 conn(0x7f4d880a24e0 legacy=0x7f4d8813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: _finish_hunting 0 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 monclient: found mon.noname-a 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d88136650 con 0x7f4d88135b60 2024-09-15T20:44:52.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: _finish_auth 0 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: _check_auth_tickets 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4d80001a00 con 0x7f4d88135b60 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: dump: 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.451 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 5 monclient: authenticate success, global_id 114351 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d80001ba0 con 0x7f4d88135b60 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 1 -- 172.21.15.78:0/3655151104 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4d80001fb0 con 0x7f4d88135b60 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8fb74640 4 set_mon_vals no callback set 2024-09-15T20:44:52.452 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.438+0000 7f4d8c90e640 10 monclient: dump: 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: shutdown 2024-09-15T20:44:52.453 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- 172.21.15.78:0/3655151104 >> v1:172.21.15.78:6789/0 conn(0x7f4d88135b60 legacy=0x7f4d88135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- 172.21.15.78:0/3655151104 shutdown_connections 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- 172.21.15.78:0/3655151104 >> 172.21.15.78:0/3655151104 conn(0x7f4d880a63c0 msgr2=0x7f4d880a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- 172.21.15.78:0/3655151104 shutdown_connections 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- 172.21.15.78:0/3655151104 wait complete. 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: build_initial_monmap 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: monmap: 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.454 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 librados: starting msgr at 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 librados: starting objecter 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 Processor -- start 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- start start 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 librados: setting wanted keys 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 librados: calling monclient init 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: init 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:52.455 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient(hunting): picked mon.noname-a con 0x7f4d880a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient(hunting): picked mon.noname-b con 0x7f4d88135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient(hunting): picked mon.noname-c con 0x7f4d8813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d881387d0 con 0x7f4d880a24e0 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d8813fa70 con 0x7f4d88135b60 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4d8813bb70 con 0x7f4d8813c220 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient(hunting): authenticate will time out at 1953.330168s 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8d910640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4d880a24e0 0x7f4d88071e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57862/0 (socket says 172.21.15.78:57862) 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8d910640 1 -- 172.21.15.78:0/2786988953 learned_addr learned my addr 172.21.15.78:0/2786988953 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 client.?.objecter ms_handle_connect 0x7f4d880a24e0 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:52.456 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 client.?.objecter ms_handle_connect 0x7f4d8813c220 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1933169634 0 0) 0x7f4d881387d0 con 0x7f4d880a24e0 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): my global_id is 114354 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d74004ec0 con 0x7f4d880a24e0 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4147864305 0 0) 0x7f4d8813bb70 con 0x7f4d8813c220 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): my global_id is 124238 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d881387d0 con 0x7f4d8813c220 2024-09-15T20:44:52.457 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 client.?.objecter ms_handle_connect 0x7f4d88135b60 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1638986506 0 0) 0x7f4d881387d0 con 0x7f4d8813c220 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d8813bb70 con 0x7f4d8813c220 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4d780012e0 con 0x7f4d8813c220 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): dump: 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.458 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 506591397 0 0) 0x7f4d74004ec0 con 0x7f4d880a24e0 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4d881387d0 con 0x7f4d880a24e0 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4d80001f70 con 0x7f4d880a24e0 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): dump: 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.459 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1340502832 0 0) 0x7f4d8813fa70 con 0x7f4d88135b60 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient(hunting): my global_id is 134125 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4d74004ec0 con 0x7f4d88135b60 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2094507957 0 0) 0x7f4d8813bb70 con 0x7f4d8813c220 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 >> v1:172.21.15.146:6789/0 conn(0x7f4d88135b60 legacy=0x7f4d88072540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 >> v1:172.21.15.78:6789/0 conn(0x7f4d880a24e0 legacy=0x7f4d88071e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient: _finish_hunting 0 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 monclient: found mon.c 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d8809d180 con 0x7f4d8813c220 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient: _finish_auth 0 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient: _check_auth_tickets 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:52.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 5 monclient: authenticate success, global_id 124238 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: _renew_subs 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4d88073690 con 0x7f4d8813c220 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 client.124238.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: _renew_subs 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4d88072c50 con 0x7f4d8813c220 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d8fb74640 1 librados: init done 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4d78001730 con 0x7f4d8813c220 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4d78004990 con 0x7f4d8813c220 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:52.462 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d87fff640 4 set_mon_vals no callback set 2024-09-15T20:44:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.442+0000 7f4d867fc640 10 monclient: dump: 2024-09-15T20:44:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.463 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d8fb74640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d8fb74640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d8fb74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d68005090 con 0x7f4d8813c220 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f4d78010a00 con 0x7f4d8813c220 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7f4d7804b790 con 0x7f4d8813c220 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d867fc640 10 client.124238.objecter ms_dispatch 0x7f4d88000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d867fc640 3 client.124238.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d867fc640 3 client.124238.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d867fc640 20 client.124238.objecter dump_active .. 0 homeless 2024-09-15T20:44:52.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d8d910640 10 monclient: get_auth_request con 0x7f4d7403e200 auth_method 0 2024-09-15T20:44:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.446+0000 7f4d867fc640 10 client.124238.objecter ms_handle_connect 0x7f4d7403e200 2024-09-15T20:44:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.450+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4d78016930 con 0x7f4d8813c220 2024-09-15T20:44:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.450+0000 7f4d867fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:52.465 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.450+0000 7f4d867fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.678+0000 7f4d8fb74640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:44:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.678+0000 7f4d8fb74640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:44:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.678+0000 7f4d8fb74640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:52.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.678+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4d68005380 con 0x7f4d8813c220 2024-09-15T20:44:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.678+0000 7f4d867fc640 1 -- 172.21.15.78:0/2786988953 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2590167838) 0x7f4d78016230 con 0x7f4d8813c220 2024-09-15T20:44:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d867fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:44:52.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d867fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:52.683 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:44:52.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 10 librados: watch_flush enter 2024-09-15T20:44:52.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 10 librados: watch_flush exit 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 20 client.124238.objecter shutdown clearing up homeless session... 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 10 client.124238.objecter successfully canceled tick 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 >> v1:172.21.15.146:6812/435598642 conn(0x7f4d7403e200 legacy=0x7f4d740406c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 10 monclient: shutdown 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 >> v1:172.21.15.78:6790/0 conn(0x7f4d8813c220 legacy=0x7f4d881d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 shutdown_connections 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 >> 172.21.15.78:0/2786988953 conn(0x7f4d880a63c0 msgr2=0x7f4d8812e9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 shutdown_connections 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 1 -- 172.21.15.78:0/2786988953 wait complete. 2024-09-15T20:44:52.686 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.682+0000 7f4d8fb74640 1 librados: shutdown 2024-09-15T20:44:52.700 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:44:52.700 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:44:52.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:44:52.701 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:44:52.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.818+0000 7fcec5441640 10 monclient: get_monmap_and_config 2024-09-15T20:44:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.818+0000 7fcec5441640 10 monclient: build_initial_monmap 2024-09-15T20:44:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.818+0000 7fcec5441640 10 monclient: monmap: 2024-09-15T20:44:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:52.822 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:52.823430+0000 2024-09-15T20:44:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:52.823430+0000 2024-09-15T20:44:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:52.823 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:52.824 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:52.824 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.818+0000 7fcec5441640 1 Processor -- start 2024-09-15T20:44:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.818+0000 7fcec5441640 1 -- start start 2024-09-15T20:44:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.818+0000 7fcec5441640 10 monclient: init 2024-09-15T20:44:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:44:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): picked mon.noname-a con 0x7fcec0137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): picked mon.noname-b con 0x7fcec0074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:52.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): picked mon.noname-c con 0x7fcec013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec0138850 con 0x7fcec0137d60 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec00a5da0 con 0x7fcec013e370 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec0073560 con 0x7fcec0074500 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): authenticate will time out at 1953.708574s 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebeffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcec0137d60 0x7fcec0138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57876/0 (socket says 172.21.15.78:57876) 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebeffd640 1 -- 172.21.15.78:0/1597730756 learned_addr learned my addr 172.21.15.78:0/1597730756 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1083678360 0 0) 0x7fcec0138850 con 0x7fcec0137d60 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient(hunting): my global_id is 114357 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcea8003530 con 0x7fcec0137d60 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1181698617 0 0) 0x7fcec0073560 con 0x7fcec0074500 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient(hunting): my global_id is 124241 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcec0138850 con 0x7fcec0074500 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 165397465 0 0) 0x7fcea8003530 con 0x7fcec0137d60 2024-09-15T20:44:52.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 >> v1:172.21.15.78:6790/0 conn(0x7fcec0074500 legacy=0x7fcec013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 >> v1:172.21.15.146:6789/0 conn(0x7fcec013e370 legacy=0x7fcec0140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: _finish_hunting 0 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 monclient: found mon.noname-a 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec0073840 con 0x7fcec0137d60 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: _finish_auth 0 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: _check_auth_tickets 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:52.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fceb00016a0 con 0x7fcec0137d60 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 5 monclient: authenticate success, global_id 114357 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: dump: 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.848 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fceb00018c0 con 0x7fcec0137d60 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 4 set_mon_vals no callback set 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 1 -- 172.21.15.78:0/1597730756 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fceb0001c90 con 0x7fcec0137d60 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebdffb640 10 monclient: dump: 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.849 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: shutdown 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- 172.21.15.78:0/1597730756 >> v1:172.21.15.78:6789/0 conn(0x7fcec0137d60 legacy=0x7fcec0138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- 172.21.15.78:0/1597730756 shutdown_connections 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- 172.21.15.78:0/1597730756 >> 172.21.15.78:0/1597730756 conn(0x7fcec00aa650 msgr2=0x7fcec00aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:52.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- 172.21.15.78:0/1597730756 shutdown_connections 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- 172.21.15.78:0/1597730756 wait complete. 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: build_initial_monmap 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: monmap: 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 librados: starting msgr at 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 librados: starting objecter 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 Processor -- start 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- start start 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 librados: setting wanted keys 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 librados: calling monclient init 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: init 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): picked mon.noname-a con 0x7fcec0074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): picked mon.noname-b con 0x7fcec0137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): picked mon.noname-c con 0x7fcec013e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec0139840 con 0x7fcec0074500 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec013b830 con 0x7fcec0137d60 2024-09-15T20:44:52.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec013bb10 con 0x7fcec013e370 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcec5441640 10 monclient(hunting): authenticate will time out at 1953.711559s 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebeffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcec0074500 0x7fcec0138a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57888/0 (socket says 172.21.15.78:57888) 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcebeffd640 1 -- 172.21.15.78:0/2734188965 learned_addr learned my addr 172.21.15.78:0/2734188965 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 client.?.objecter ms_handle_connect 0x7fcec013e370 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 client.?.objecter ms_handle_connect 0x7fcec0074500 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 742804211 0 0) 0x7fcec013bb10 con 0x7fcec013e370 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 monclient(hunting): my global_id is 124244 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceac004ec0 con 0x7fcec013e370 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3410887071 0 0) 0x7fcec0139840 con 0x7fcec0074500 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 10 monclient(hunting): my global_id is 114360 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.822+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcec013bb10 con 0x7fcec0074500 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 client.?.objecter ms_handle_connect 0x7fcec0137d60 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3185304258 0 0) 0x7fcec013bb10 con 0x7fcec0074500 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcec0139840 con 0x7fcec0074500 2024-09-15T20:44:52.854 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fceb0001c10 con 0x7fcec0074500 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): dump: 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.855 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4288773761 0 0) 0x7fceac004ec0 con 0x7fcec013e370 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcec013bb10 con 0x7fcec013e370 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fceb8001c40 con 0x7fcec013e370 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): dump: 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.856 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 825246315 0 0) 0x7fcec013b830 con 0x7fcec0137d60 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): my global_id is 134131 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fceac004ec0 con 0x7fcec0137d60 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4124419779 0 0) 0x7fcec0139840 con 0x7fcec0074500 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 >> v1:172.21.15.78:6790/0 conn(0x7fcec013e370 legacy=0x7fcec01d5010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 >> v1:172.21.15.146:6789/0 conn(0x7fcec0137d60 legacy=0x7fcec0139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.857 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 monclient: found mon.a 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec013ab30 con 0x7fcec0074500 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: _finish_auth 0 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 5 monclient: authenticate success, global_id 114360 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 10 monclient: _renew_subs 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:52.858 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcec013c4f0 con 0x7fcec0074500 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 10 client.114360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 10 monclient: _renew_subs 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fcec013da00 con 0x7fcec0074500 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 1 librados: init done 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fceb0001820 con 0x7fcec0074500 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fceb0004da0 con 0x7fcec0074500 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcebd7fa640 4 set_mon_vals no callback set 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: dump: 2024-09-15T20:44:52.859 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcea00065d0 con 0x7fcea0002870 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 client.114360.objecter ms_handle_connect 0x7fcea0002870 2024-09-15T20:44:52.860 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 client.114360.objecter resend_mon_ops 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3602763386 0 0) 0x7fcea00065d0 con 0x7fcea0002870 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient(hunting): my global_id is 114363 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcec0139840 con 0x7fcea0002870 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fceb0010e10 con 0x7fcec0074500 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7fceb004ba10 con 0x7fcec0074500 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 client.114360.objecter ms_dispatch 0x7fcec0000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 3 client.114360.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 3 client.114360.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 20 client.114360.objecter dump_active .. 0 homeless 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2970939061 0 0) 0x7fcec0139840 con 0x7fcea0002870 2024-09-15T20:44:52.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcea00065d0 con 0x7fcea0002870 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fceb4001750 con 0x7fcea0002870 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: dump: 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.862 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4088124658 0 0) 0x7fcea00065d0 con 0x7fcea0002870 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcea000a170 con 0x7fcea0002870 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcebf7fe640 10 monclient: get_auth_request con 0x7fceac03e1b0 auth_method 0 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fceb4001910 con 0x7fcea0002870 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 >> v1:172.21.15.78:6789/0 conn(0x7fcea0002870 legacy=0x7fcea0004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 >> v1:172.21.15.78:6789/0 conn(0x7fcea0002870 legacy=0x7fcea0004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.826+0000 7fcea77fe640 10 client.114360.objecter ms_handle_connect 0x7fceac03e1b0 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:44:52.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fcea0009e80 con 0x7fcea0009140 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 client.114360.objecter ms_handle_connect 0x7fcea0009140 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 client.114360.objecter resend_mon_ops 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2726365590 0 0) 0x7fcea0009e80 con 0x7fcea0009140 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient(hunting): my global_id is 114366 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcea00065d0 con 0x7fcea0009140 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3543923813 0 0) 0x7fcea00065d0 con 0x7fcea0009140 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcea0009e80 con 0x7fcea0009140 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fceb40037c0 con 0x7fcea0009140 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:52.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient: dump: 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 584532055 0 0) 0x7fcea0009e80 con 0x7fcea0009140 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcea000c420 con 0x7fcea0009140 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4307 (unknown 0 0 2643265901) 0x7fceb4003750 con 0x7fcea0009140 2024-09-15T20:44:52.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 >> v1:172.21.15.78:6789/0 conn(0x7fcea0009140 legacy=0x7fcea00095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcea77fe640 1 -- 172.21.15.78:0/2734188965 >> v1:172.21.15.78:6789/0 conn(0x7fcea0009140 legacy=0x7fcea00095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 54, 2024-09-15T20:44:52.866 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 218677, 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:52.867 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:52.868 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:44:52.869 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:44:52.870 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:52.871 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.872 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:44:52.873 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:44:52.874 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:52.875 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:44:52.876 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:44:52.877 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:52.878 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:52.879 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:52.880 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:44:52.881 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:44:52.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 10 librados: watch_flush enter 2024-09-15T20:44:52.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 10 librados: watch_flush exit 2024-09-15T20:44:52.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 20 client.114360.objecter shutdown clearing up homeless session... 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 10 client.114360.objecter successfully canceled tick 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 >> v1:172.21.15.146:6812/435598642 conn(0x7fceac03e1b0 legacy=0x7fceac040670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 10 monclient: shutdown 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 >> v1:172.21.15.78:6789/0 conn(0x7fcec0074500 legacy=0x7fcec0138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcebf7fe640 1 -- 172.21.15.78:0/2734188965 reap_dead start 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcebf7fe640 1 -- 172.21.15.78:0/2734188965 reap_dead start 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 shutdown_connections 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 >> 172.21.15.78:0/2734188965 conn(0x7fcec00aa650 msgr2=0x7fcec00a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 shutdown_connections 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 1 -- 172.21.15.78:0/2734188965 wait complete. 2024-09-15T20:44:52.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:52.834+0000 7fcec5441640 1 librados: shutdown 2024-09-15T20:44:52.884 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:44:52.884 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:44:52.884 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:44:53.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient: get_monmap_and_config 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient: build_initial_monmap 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient: monmap: 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:53.005378+0000 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:53.005378+0000 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:53.004 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:53.005 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 Processor -- start 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- start start 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient: init 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient(hunting): picked mon.noname-a con 0x7fbff80a0ff0 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient(hunting): picked mon.noname-c con 0x7fbff8073800 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient(hunting): picked mon.noname-b con 0x7fbff80a1ae0 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:53.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff8072df0 con 0x7fbff80a0ff0 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff8074110 con 0x7fbff8073800 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff80743f0 con 0x7fbff80a1ae0 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient(hunting): authenticate will time out at 1953.890186s 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffc9f0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbff80a0ff0 0x7fbff80a13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57898/0 (socket says 172.21.15.78:57898) 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffc9f0640 1 -- 172.21.15.78:0/1317690645 learned_addr learned my addr 172.21.15.78:0/1317690645 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 961216414 0 0) 0x7fbff80743f0 con 0x7fbff80a1ae0 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient(hunting): my global_id is 124247 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe0003530 con 0x7fbff80a1ae0 2024-09-15T20:44:53.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3930393017 0 0) 0x7fbff8072df0 con 0x7fbff80a0ff0 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient(hunting): my global_id is 114369 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff80743f0 con 0x7fbff80a0ff0 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2374763155 0 0) 0x7fbfe0003530 con 0x7fbff80a1ae0 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 >> v1:172.21.15.146:6789/0 conn(0x7fbff8073800 legacy=0x7fbff813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 >> v1:172.21.15.78:6789/0 conn(0x7fbff80a0ff0 legacy=0x7fbff80a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:44:53.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 monclient: found mon.noname-b 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff8074eb0 con 0x7fbff80a1ae0 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: _finish_auth 0 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbff0001950 con 0x7fbff80a1ae0 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: dump: 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.017 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 5 monclient: authenticate success, global_id 124247 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbff0001af0 con 0x7fbff80a1ae0 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 1 -- 172.21.15.78:0/1317690645 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbff0001f00 con 0x7fbff80a1ae0 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:53.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbff77fe640 10 monclient: dump: 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 4 set_mon_vals no callback set 2024-09-15T20:44:53.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 10 monclient: shutdown 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- 172.21.15.78:0/1317690645 >> v1:172.21.15.78:6790/0 conn(0x7fbff80a1ae0 legacy=0x7fbff8140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- 172.21.15.78:0/1317690645 shutdown_connections 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- 172.21.15.78:0/1317690645 >> 172.21.15.78:0/1317690645 conn(0x7fbff80a63c0 msgr2=0x7fbff80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- 172.21.15.78:0/1317690645 shutdown_connections 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.002+0000 7fbffec54640 1 -- 172.21.15.78:0/1317690645 wait complete. 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: build_initial_monmap 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: monmap: 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:53.020 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 librados: starting msgr at 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 librados: starting objecter 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 Processor -- start 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 -- start start 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 librados: setting wanted keys 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 librados: calling monclient init 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: init 2024-09-15T20:44:53.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient(hunting): picked mon.noname-a con 0x7fbff8073800 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient(hunting): picked mon.noname-b con 0x7fbff80a0ff0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient(hunting): picked mon.noname-c con 0x7fbff80a1ae0 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff8137370 con 0x7fbff8073800 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff8138ac0 con 0x7fbff80a0ff0 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbff809d9e0 con 0x7fbff80a1ae0 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient(hunting): authenticate will time out at 1953.893422s 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffc9f0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbff8073800 0x7fbff8136550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57900/0 (socket says 172.21.15.78:57900) 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffc9f0640 1 -- 172.21.15.78:0/1841242828 learned_addr learned my addr 172.21.15.78:0/1841242828 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.?.objecter ms_handle_connect 0x7fbff8073800 2024-09-15T20:44:53.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.?.objecter ms_handle_connect 0x7fbff80a1ae0 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1000063198 0 0) 0x7fbff8137370 con 0x7fbff8073800 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): my global_id is 114372 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfe4004ec0 con 0x7fbff8073800 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3338281241 0 0) 0x7fbff809d9e0 con 0x7fbff80a1ae0 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): my global_id is 124250 2024-09-15T20:44:53.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff8137370 con 0x7fbff80a1ae0 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.?.objecter ms_handle_connect 0x7fbff80a0ff0 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3703684805 0 0) 0x7fbfe4004ec0 con 0x7fbff8073800 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbff809d9e0 con 0x7fbff8073800 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbfe80013c0 con 0x7fbff8073800 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): dump: 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.024 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2664545307 0 0) 0x7fbff8137370 con 0x7fbff80a1ae0 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfe4004ec0 con 0x7fbff80a1ae0 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbff0001810 con 0x7fbff80a1ae0 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): dump: 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.025 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 586891702 0 0) 0x7fbff8138ac0 con 0x7fbff80a0ff0 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): my global_id is 134134 2024-09-15T20:44:53.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff8137370 con 0x7fbff80a0ff0 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3051572935 0 0) 0x7fbff809d9e0 con 0x7fbff8073800 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 >> v1:172.21.15.78:6790/0 conn(0x7fbff80a1ae0 legacy=0x7fbff81d4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 >> v1:172.21.15.146:6789/0 conn(0x7fbff80a0ff0 legacy=0x7fbff8136c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: _finish_hunting 0 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 monclient: found mon.a 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff809d180 con 0x7fbff8073800 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: _finish_auth 0 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: _check_auth_tickets 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 5 monclient: authenticate success, global_id 114372 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: _renew_subs 2024-09-15T20:44:53.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbff813a5c0 con 0x7fbff8073800 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 client.114372.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: _renew_subs 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbff8141810 con 0x7fbff8073800 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 librados: init done 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbfe8001680 con 0x7fbff8073800 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbfe8004c40 con 0x7fbff8073800 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff6ffd640 4 set_mon_vals no callback set 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:53.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: dump: 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.029 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.030 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:53.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbfd80065d0 con 0x7fbfd8002870 2024-09-15T20:44:53.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.114372.objecter ms_handle_connect 0x7fbfd8002870 2024-09-15T20:44:53.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.114372.objecter resend_mon_ops 2024-09-15T20:44:53.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 703501589 0 0) 0x7fbfd80065d0 con 0x7fbfd8002870 2024-09-15T20:44:53.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient(hunting): my global_id is 124253 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbff809d9e0 con 0x7fbfd8002870 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fbfe8010cb0 con 0x7fbff8073800 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7fbfe804ba70 con 0x7fbff8073800 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 client.114372.objecter ms_dispatch 0x7fbff8000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 3 client.114372.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 3 client.114372.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 20 client.114372.objecter dump_active .. 0 homeless 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1082854161 0 0) 0x7fbff809d9e0 con 0x7fbfd8002870 2024-09-15T20:44:53.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfd80065d0 con 0x7fbfd8002870 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbfec001680 con 0x7fbfd8002870 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 10 monclient: dump: 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.032 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4052726166 0 0) 0x7fbfd80065d0 con 0x7fbfd8002870 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.006+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbfd800a170 con 0x7fbfd8002870 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.010+0000 7fbffd1f1640 10 monclient: get_auth_request con 0x7fbfe403e1c0 auth_method 0 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.010+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbfec001960 con 0x7fbfd8002870 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.010+0000 7fbff57fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.010+0000 7fbff57fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.010+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 >> v1:172.21.15.78:6790/0 conn(0x7fbfd8002870 legacy=0x7fbfd8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.010+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 >> v1:172.21.15.78:6790/0 conn(0x7fbfd8002870 legacy=0x7fbfd8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:53.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.010+0000 7fbff57fa640 10 client.114372.objecter ms_handle_connect 0x7fbfe403e1c0 2024-09-15T20:44:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:44:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbfd8009e80 con 0x7fbfd8009140 2024-09-15T20:44:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 client.114372.objecter ms_handle_connect 0x7fbfd8009140 2024-09-15T20:44:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 client.114372.objecter resend_mon_ops 2024-09-15T20:44:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1568478500 0 0) 0x7fbfd8009e80 con 0x7fbfd8009140 2024-09-15T20:44:53.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient(hunting): my global_id is 124256 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbfd80065d0 con 0x7fbfd8009140 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4222314545 0 0) 0x7fbfd80065d0 con 0x7fbfd8009140 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbfd8009e80 con 0x7fbfd8009140 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbfec0037a0 con 0x7fbfd8009140 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient: dump: 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.036 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4232229409 0 0) 0x7fbfd8009e80 con 0x7fbfd8009140 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbfd800c400 con 0x7fbfd8009140 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 3032078342) 0x7fbfec003730 con 0x7fbfd8009140 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 >> v1:172.21.15.78:6790/0 conn(0x7fbfd8009140 legacy=0x7fbfd80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbff57fa640 1 -- 172.21.15.78:0/1841242828 >> v1:172.21.15.78:6790/0 conn(0x7fbfd8009140 legacy=0x7fbfd80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 54, 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 110942, 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:44:53.038 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:53.039 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:53.040 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:44:53.041 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:44:53.042 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.043 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.044 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.045 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:44:53.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:53.047 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:44:53.048 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.049 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:44:53.050 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:44:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 10 librados: watch_flush enter 2024-09-15T20:44:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 10 librados: watch_flush exit 2024-09-15T20:44:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 20 client.114372.objecter shutdown clearing up homeless session... 2024-09-15T20:44:53.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 10 client.114372.objecter successfully canceled tick 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 >> v1:172.21.15.146:6812/435598642 conn(0x7fbfe403e1c0 legacy=0x7fbfe4040680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 10 monclient: shutdown 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 >> v1:172.21.15.78:6789/0 conn(0x7fbff8073800 legacy=0x7fbff8136550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffd1f1640 1 -- 172.21.15.78:0/1841242828 reap_dead start 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffd1f1640 1 -- 172.21.15.78:0/1841242828 reap_dead start 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 shutdown_connections 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 >> 172.21.15.78:0/1841242828 conn(0x7fbff80a63c0 msgr2=0x7fbff8138170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 shutdown_connections 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 1 -- 172.21.15.78:0/1841242828 wait complete. 2024-09-15T20:44:53.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.014+0000 7fbffec54640 1 librados: shutdown 2024-09-15T20:44:53.053 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:44:53.053 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:44:53.053 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: get_monmap_and_config 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: build_initial_monmap 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: monmap: 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:53.168981+0000 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:53.168981+0000 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:53.177 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.178 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:53.178 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:53.178 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:53.178 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 Processor -- start 2024-09-15T20:44:53.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- start start 2024-09-15T20:44:53.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: init 2024-09-15T20:44:53.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient(hunting): picked mon.noname-c con 0x7f5ffc137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient(hunting): picked mon.noname-b con 0x7f5ffc0a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient(hunting): picked mon.noname-a con 0x7f5ffc13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ffc0a6420 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ffc0a6700 con 0x7f5ffc137ce0 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ffc0a2110 con 0x7f5ffc0a9610 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient(hunting): authenticate will time out at 1954.053529s 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ffb7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5ffc13e5f0 0x7f5ffc1409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57910/0 (socket says 172.21.15.78:57910) 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ffa7fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f5ffc0a9610 0x7f5ffc13ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56648/0 (socket says 172.21.15.78:56648) 2024-09-15T20:44:53.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ffb7fe640 1 -- 172.21.15.78:0/1724540238 learned_addr learned my addr 172.21.15.78:0/1724540238 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:53.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3109318152 0 0) 0x7f5ffc0a2110 con 0x7f5ffc0a9610 2024-09-15T20:44:53.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient(hunting): my global_id is 124259 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fdc003530 con 0x7f5ffc0a9610 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2492721093 0 0) 0x7f5ffc0a6420 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient(hunting): my global_id is 114375 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ffc0a2110 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3561001215 0 0) 0x7f5fdc003530 con 0x7f5ffc0a9610 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 >> v1:172.21.15.146:6789/0 conn(0x7f5ffc137ce0 legacy=0x7f5ffc1380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 >> v1:172.21.15.78:6789/0 conn(0x7f5ffc13e5f0 legacy=0x7f5ffc1409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 monclient: found mon.noname-b 2024-09-15T20:44:53.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ffc0a4bc0 con 0x7f5ffc0a9610 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: _finish_auth 0 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5fe40016a0 con 0x7f5ffc0a9610 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 5 monclient: authenticate success, global_id 124259 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: dump: 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.182 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5fe40018c0 con 0x7f5ffc0a9610 2024-09-15T20:44:53.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 1 -- 172.21.15.78:0/1724540238 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5fe4001cd0 con 0x7f5ffc0a9610 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f5ff9ffb640 10 monclient: dump: 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.184 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 4 set_mon_vals no callback set 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: shutdown 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- 172.21.15.78:0/1724540238 >> v1:172.21.15.78:6790/0 conn(0x7f5ffc0a9610 legacy=0x7f5ffc13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- 172.21.15.78:0/1724540238 shutdown_connections 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- 172.21.15.78:0/1724540238 >> 172.21.15.78:0/1724540238 conn(0x7f5ffc0aa6c0 msgr2=0x7f5ffc0aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- 172.21.15.78:0/1724540238 shutdown_connections 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 -- 172.21.15.78:0/1724540238 wait complete. 2024-09-15T20:44:53.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: build_initial_monmap 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 10 monclient: monmap: 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 librados: starting msgr at 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.166+0000 7f6001863640 1 librados: starting objecter 2024-09-15T20:44:53.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 Processor -- start 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 -- start start 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 librados: setting wanted keys 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 librados: calling monclient init 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient: init 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient(hunting): picked mon.noname-b con 0x7f5ffc0a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient(hunting): picked mon.noname-c con 0x7f5ffc137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient(hunting): picked mon.noname-a con 0x7f5ffc13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ffc074330 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ffc13c0e0 con 0x7f5ffc0a9610 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ffc13ab00 con 0x7f5ffc137ce0 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient(hunting): authenticate will time out at 1954.056625s 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ffa7fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f5ffc137ce0 0x7f5ffc1d1930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56650/0 (socket says 172.21.15.78:56650) 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ffb7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5ffc13e5f0 0x7f5ffc1d5060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57918/0 (socket says 172.21.15.78:57918) 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ffa7fc640 1 -- 172.21.15.78:0/3422691166 learned_addr learned my addr 172.21.15.78:0/3422691166 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.?.objecter ms_handle_connect 0x7f5ffc137ce0 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.?.objecter ms_handle_connect 0x7f5ffc13e5f0 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3683372449 0 0) 0x7f5ffc13ab00 con 0x7f5ffc137ce0 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.188 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): my global_id is 124262 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fe0004ec0 con 0x7f5ffc137ce0 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952347384 0 0) 0x7f5fec001830 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): my global_id is 114378 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ffc13ab00 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.?.objecter ms_handle_connect 0x7f5ffc0a9610 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1990030893 0 0) 0x7f5fe0004ec0 con 0x7f5ffc137ce0 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fec001830 con 0x7f5ffc137ce0 2024-09-15T20:44:53.189 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5fe4001cd0 con 0x7f5ffc137ce0 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): dump: 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:53.190 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3445515456 0 0) 0x7f5ffc13ab00 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fe0004ec0 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5fec001190 con 0x7f5ffc13e5f0 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): dump: 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.191 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3150083856 0 0) 0x7f5ffc13c0e0 con 0x7f5ffc0a9610 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): my global_id is 134140 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ffc13ab00 con 0x7f5ffc0a9610 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 979657847 0 0) 0x7f5fec001830 con 0x7f5ffc137ce0 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 >> v1:172.21.15.146:6789/0 conn(0x7f5ffc0a9610 legacy=0x7f5ffc1c94f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.192 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 >> v1:172.21.15.78:6789/0 conn(0x7f5ffc13e5f0 legacy=0x7f5ffc1d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: _finish_hunting 0 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 monclient: found mon.c 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ffc0a91e0 con 0x7f5ffc137ce0 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: _finish_auth 0 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: _check_auth_tickets 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 5 monclient: authenticate success, global_id 124262 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient: _renew_subs 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5ffc13be00 con 0x7f5ffc137ce0 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 client.124262.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:53.193 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient: _renew_subs 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5ffc0a0230 con 0x7f5ffc137ce0 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 librados: init done 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5fe40018c0 con 0x7f5ffc137ce0 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5fe4004aa0 con 0x7f5ffc137ce0 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff97fa640 4 set_mon_vals no callback set 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: dump: 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.194 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fd40065d0 con 0x7f5fd4002870 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f5fe4010b90 con 0x7f5ffc137ce0 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.124262.objecter ms_handle_connect 0x7f5fd4002870 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.124262.objecter resend_mon_ops 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7f5fe404b8d0 con 0x7f5ffc137ce0 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.124262.objecter ms_dispatch 0x7f5ffc000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:53.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 3 client.124262.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 3 client.124262.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 20 client.124262.objecter dump_active .. 0 homeless 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2968641841 0 0) 0x7f5fd40065d0 con 0x7f5fd4002870 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient(hunting): my global_id is 134143 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fec001830 con 0x7f5fd4002870 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ffb7fe640 10 monclient: get_auth_request con 0x7f5fe003e240 auth_method 0 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 client.124262.objecter ms_handle_connect 0x7f5fe003e240 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4170264514 0 0) 0x7f5fec001830 con 0x7f5fd4002870 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fd40065d0 con 0x7f5fd4002870 2024-09-15T20:44:53.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5ff4001ba0 con 0x7f5fd4002870 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 10 monclient: dump: 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 182245155 0 0) 0x7f5fd40065d0 con 0x7f5fd4002870 2024-09-15T20:44:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.170+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5fd400a170 con 0x7f5fd4002870 2024-09-15T20:44:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.174+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f5ff4001ee0 con 0x7f5fd4002870 2024-09-15T20:44:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.174+0000 7f5ff3fff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.174+0000 7f5ff3fff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.174+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 >> v1:172.21.15.146:6789/0 conn(0x7f5fd4002870 legacy=0x7f5fd4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.174+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 >> v1:172.21.15.146:6789/0 conn(0x7f5fd4002870 legacy=0x7f5fd4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:53.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f6001863640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5fd4009e80 con 0x7f5fd4009140 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f5ff3fff640 10 client.124262.objecter ms_handle_connect 0x7f5fd4009140 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f5ff3fff640 10 client.124262.objecter resend_mon_ops 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2200763778 0 0) 0x7f5fd4009e80 con 0x7f5fd4009140 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f5ff3fff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f5ff3fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f5ff3fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f5ff3fff640 10 monclient(hunting): my global_id is 134146 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.178+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fd40065d0 con 0x7f5fd4009140 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 954085471 0 0) 0x7f5fd40065d0 con 0x7f5fd4009140 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5fd4009e80 con 0x7f5fd4009140 2024-09-15T20:44:53.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5ff4003b70 con 0x7f5fd4009140 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 10 monclient: dump: 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.201 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4001349739 0 0) 0x7f5fd4009e80 con 0x7f5fd4009140 2024-09-15T20:44:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5fd400c420 con 0x7f5fd4009140 2024-09-15T20:44:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3969 (unknown 0 0 1263190870) 0x7f5ff4003b70 con 0x7f5fd4009140 2024-09-15T20:44:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:44:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 >> v1:172.21.15.146:6789/0 conn(0x7f5fd4009140 legacy=0x7f5fd40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ff3fff640 1 -- 172.21.15.78:0/3422691166 >> v1:172.21.15.146:6789/0 conn(0x7f5fd4009140 legacy=0x7f5fd40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:44:53.202 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 54, 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 3051, 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:44:53.203 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:53.204 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:53.205 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:44:53.206 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:44:53.207 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.208 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:53.209 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:44:53.210 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:53.211 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:44:53.212 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.213 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:44:53.214 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:44:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:44:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:44:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:44:53.215 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:44:53.215 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:44:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 10 librados: watch_flush enter 2024-09-15T20:44:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 10 librados: watch_flush exit 2024-09-15T20:44:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 20 client.124262.objecter shutdown clearing up homeless session... 2024-09-15T20:44:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 10 client.124262.objecter successfully canceled tick 2024-09-15T20:44:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 >> v1:172.21.15.146:6812/435598642 conn(0x7f5fe003e240 legacy=0x7f5fe0040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 10 monclient: shutdown 2024-09-15T20:44:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:53.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 >> v1:172.21.15.78:6790/0 conn(0x7f5ffc137ce0 legacy=0x7f5ffc1d1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ffb7fe640 1 -- 172.21.15.78:0/3422691166 reap_dead start 2024-09-15T20:44:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f5ffb7fe640 1 -- 172.21.15.78:0/3422691166 reap_dead start 2024-09-15T20:44:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 shutdown_connections 2024-09-15T20:44:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 >> 172.21.15.78:0/3422691166 conn(0x7f5ffc0aa6c0 msgr2=0x7f5ffc073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 shutdown_connections 2024-09-15T20:44:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 1 -- 172.21.15.78:0/3422691166 wait complete. 2024-09-15T20:44:53.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.182+0000 7f6001863640 1 librados: shutdown 2024-09-15T20:44:53.218 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:44:53.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-09-15T20:44:53.219 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T20:44:53.219 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-09-15T20:44:53.219 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-09-15T20:44:53.219 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T20:44:53.219 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:44:53.232 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T20:44:53.233 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:44:53.233 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:44:53.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: get_monmap_and_config 2024-09-15T20:44:53.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: build_initial_monmap 2024-09-15T20:44:53.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: monmap: 2024-09-15T20:44:53.375 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:53.375 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.376 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:53.377319+0000 2024-09-15T20:44:53.376 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:53.377319+0000 2024-09-15T20:44:53.376 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:53.376 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.376 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:53.376 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:53.376 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:53.376 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 Processor -- start 2024-09-15T20:44:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- start start 2024-09-15T20:44:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: init 2024-09-15T20:44:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:44:53.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient(hunting): picked mon.noname-a con 0x7f1578074630 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient(hunting): picked mon.noname-b con 0x7f157809d950 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient(hunting): picked mon.noname-c con 0x7f1578074a10 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1578134bc0 con 0x7f1578074630 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f157809f900 con 0x7f1578074a10 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f157809fbe0 con 0x7f157809d950 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient(hunting): authenticate will time out at 1954.261859s 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157d0c9640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f1578074630 legacy=0x7f1578133da0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:44:53.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157c8c8640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f157809d950 0x7f15781344b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56680/0 (socket says 172.21.15.78:56680) 2024-09-15T20:44:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157c8c8640 1 -- 172.21.15.78:0/1392712230 learned_addr learned my addr 172.21.15.78:0/1392712230 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 358808687 0 0) 0x7f157809fbe0 con 0x7f157809d950 2024-09-15T20:44:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient(hunting): my global_id is 124265 2024-09-15T20:44:53.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1564003530 con 0x7f157809d950 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 518637022 0 0) 0x7f1564003530 con 0x7f157809d950 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 >> v1:172.21.15.146:6789/0 conn(0x7f1578074a10 legacy=0x7f157813d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 >> v1:172.21.15.78:6789/0 conn(0x7f1578074630 legacy=0x7f1578133da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 monclient: found mon.noname-b 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157809fec0 con 0x7f157809d950 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: _finish_auth 0 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f15740016a0 con 0x7f157809d950 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 5 monclient: authenticate success, global_id 124265 2024-09-15T20:44:53.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: dump: 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.386 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f15740018c0 con 0x7f157809d950 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 1 -- 172.21.15.78:0/1392712230 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1574001cd0 con 0x7f157809d950 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 4 set_mon_vals no callback set 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f156ffff640 10 monclient: dump: 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.387 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: shutdown 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- 172.21.15.78:0/1392712230 >> v1:172.21.15.78:6790/0 conn(0x7f157809d950 legacy=0x7f15781344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- 172.21.15.78:0/1392712230 shutdown_connections 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- 172.21.15.78:0/1392712230 >> 172.21.15.78:0/1392712230 conn(0x7f15780a63c0 msgr2=0x7f15780a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- 172.21.15.78:0/1392712230 shutdown_connections 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 1 -- 172.21.15.78:0/1392712230 wait complete. 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: build_initial_monmap 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.374+0000 7f157f32d640 10 monclient: monmap: 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:53.388 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 librados: starting msgr at 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 librados: starting objecter 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 Processor -- start 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 -- start start 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 librados: setting wanted keys 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 librados: calling monclient init 2024-09-15T20:44:53.389 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: init 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient(hunting): picked mon.noname-a con 0x7f1578074630 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient(hunting): picked mon.noname-c con 0x7f1578074a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient(hunting): picked mon.noname-b con 0x7f157809d950 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1578136800 con 0x7f1578074630 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1578136ae0 con 0x7f157809d950 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157d0c9640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f1578074630 legacy=0x7f157813d040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f15781352e0 con 0x7f1578074a10 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient(hunting): authenticate will time out at 1954.264710s 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157c8c8640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1578074a10 0x7f15781ceb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56686/0 (socket says 172.21.15.78:56686) 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157c8c8640 1 -- 172.21.15.78:0/4262488684 learned_addr learned my addr 172.21.15.78:0/4262488684 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:53.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 client.?.objecter ms_handle_connect 0x7f1578074a10 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2744805815 0 0) 0x7f15781352e0 con 0x7f1578074a10 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient(hunting): my global_id is 124268 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f155c0070d0 con 0x7f1578074a10 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 27696506 0 0) 0x7f155c0070d0 con 0x7f1578074a10 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f15781352e0 con 0x7f1578074a10 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 client.?.objecter ms_handle_connect 0x7f157809d950 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1574001c10 con 0x7f1578074a10 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:53.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient(hunting): dump: 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3128198079 0 0) 0x7f15781352e0 con 0x7f1578074a10 2024-09-15T20:44:53.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 >> v1:172.21.15.146:6789/0 conn(0x7f157809d950 legacy=0x7f15781d2270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 >> v1:172.21.15.78:6789/0 conn(0x7f1578074630 legacy=0x7f157813d040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient: _finish_hunting 0 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 monclient: found mon.c 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f157809d180 con 0x7f1578074a10 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient: _finish_auth 0 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient: _check_auth_tickets 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 5 monclient: authenticate success, global_id 124268 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: _renew_subs 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f157812e970 con 0x7f1578074a10 2024-09-15T20:44:53.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 client.124268.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: _renew_subs 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f15781c9bf0 con 0x7f1578074a10 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 librados: init done 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1574001860 con 0x7f1578074a10 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1574004da0 con 0x7f1578074a10 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156f7fe640 4 set_mon_vals no callback set 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 monclient: dump: 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.394 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1558005090 con 0x7f1578074a10 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1574010e10 con 0x7f1578074a10 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7f157404b9e0 con 0x7f1578074a10 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 client.124268.objecter ms_dispatch 0x7f1578000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:53.395 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 3 client.124268.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 3 client.124268.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 20 client.124268.objecter dump_active .. 0 homeless 2024-09-15T20:44:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f157d0c9640 10 monclient: get_auth_request con 0x7f155c040180 auth_method 0 2024-09-15T20:44:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.378+0000 7f156dffb640 10 client.124268.objecter ms_handle_connect 0x7f155c040180 2024-09-15T20:44:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.382+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1574016b20 con 0x7f1578074a10 2024-09-15T20:44:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.382+0000 7f156dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:53.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.382+0000 7f156dffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:53.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.610+0000 7f157f32d640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:44:53.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.610+0000 7f157f32d640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:44:53.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.610+0000 7f157f32d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:53.613 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.610+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1558005380 con 0x7f1578074a10 2024-09-15T20:44:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.610+0000 7f156dffb640 1 -- 172.21.15.78:0/4262488684 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 1732305180) 0x7f1574016420 con 0x7f1578074a10 2024-09-15T20:44:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.610+0000 7f156dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:44:53.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.610+0000 7f156dffb640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:53.614 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 10 librados: watch_flush enter 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 10 librados: watch_flush exit 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 20 client.124268.objecter shutdown clearing up homeless session... 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 10 client.124268.objecter successfully canceled tick 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 >> v1:172.21.15.146:6812/435598642 conn(0x7f155c040180 legacy=0x7f155c042640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 10 monclient: shutdown 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 >> v1:172.21.15.78:6790/0 conn(0x7f1578074a10 legacy=0x7f15781ceb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 shutdown_connections 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 >> 172.21.15.78:0/4262488684 conn(0x7f15780a63c0 msgr2=0x7f1578131020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:53.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 shutdown_connections 2024-09-15T20:44:53.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 1 -- 172.21.15.78:0/4262488684 wait complete. 2024-09-15T20:44:53.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:53.614+0000 7f157f32d640 1 librados: shutdown 2024-09-15T20:44:56.632 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:44:56.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient: get_monmap_and_config 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient: build_initial_monmap 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient: monmap: 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:44:56.763462+0000 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:44:56.763462+0000 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:56.762 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:44:56.763 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:44:56.763 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:56.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 1 Processor -- start 2024-09-15T20:44:56.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 1 -- start start 2024-09-15T20:44:56.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient: init 2024-09-15T20:44:56.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:56.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:44:56.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient(hunting): picked mon.noname-b con 0x7f674c135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:56.770 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient(hunting): picked mon.noname-c con 0x7f674c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.758+0000 7f6754304640 10 monclient(hunting): picked mon.noname-a con 0x7f674c13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674c0745e0 con 0x7f674c13c220 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674c06ecf0 con 0x7f674c0a24e0 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674c06efd0 con 0x7f674c135b60 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient(hunting): authenticate will time out at 1957.648056s 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f67528a1640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f674c13c220 legacy=0x7f674c13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f67520a0640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f674c135b60 0x7f674c135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56694/0 (socket says 172.21.15.78:56694) 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f67520a0640 1 -- 172.21.15.78:0/1628371157 learned_addr learned my addr 172.21.15.78:0/1628371157 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877445306 0 0) 0x7f674c06efd0 con 0x7f674c135b60 2024-09-15T20:44:56.771 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient(hunting): _negotiate 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient(hunting): my global_id is 124271 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6734003530 con 0x7f674c135b60 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 912443505 0 0) 0x7f6734003530 con 0x7f674c135b60 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 >> v1:172.21.15.146:6789/0 conn(0x7f674c0a24e0 legacy=0x7f674c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 >> v1:172.21.15.78:6789/0 conn(0x7f674c13c220 legacy=0x7f674c13e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: _finish_hunting 0 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 monclient: found mon.noname-b 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:56.772 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c136650 con 0x7f674c135b60 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: _finish_auth 0 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: _check_auth_tickets 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 5 monclient: authenticate success, global_id 124271 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6744001a20 con 0x7f674c135b60 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: dump: 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:56.773 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6744001c40 con 0x7f674c135b60 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:56.774 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: get_monmap_and_config success 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 1 -- 172.21.15.78:0/1628371157 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6744002050 con 0x7f674c135b60 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 4 set_mon_vals no callback set 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675109e640 10 monclient: dump: 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:56.775 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: shutdown 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- 172.21.15.78:0/1628371157 >> v1:172.21.15.78:6790/0 conn(0x7f674c135b60 legacy=0x7f674c135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- 172.21.15.78:0/1628371157 shutdown_connections 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- 172.21.15.78:0/1628371157 >> 172.21.15.78:0/1628371157 conn(0x7f674c0a63c0 msgr2=0x7f674c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- 172.21.15.78:0/1628371157 shutdown_connections 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- 172.21.15.78:0/1628371157 wait complete. 2024-09-15T20:44:56.776 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: build_initial_monmap 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: monmap: 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:56.777 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 librados: starting msgr at 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 librados: starting objecter 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 Processor -- start 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- start start 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 librados: setting wanted keys 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 librados: calling monclient init 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: init 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: _reopen_session rank -1 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient(hunting): picked mon.noname-b con 0x7f674c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient(hunting): picked mon.noname-a con 0x7f674c135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient(hunting): picked mon.noname-c con 0x7f674c13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:44:56.778 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674c13fb20 con 0x7f674c135b60 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674c09e5b0 con 0x7f674c0a24e0 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f674c09dd10 con 0x7f674c13c220 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient(hunting): _renew_subs 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient(hunting): authenticate will time out at 1957.650937s 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f675189f640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7f674c135b60 legacy=0x7f674c0721a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f67528a1640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f674c13c220 0x7f674c1d4c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56696/0 (socket says 172.21.15.78:56696) 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f67528a1640 1 -- 172.21.15.78:0/439057552 learned_addr learned my addr 172.21.15.78:0/439057552 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 client.?.objecter ms_handle_connect 0x7f674c13c220 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2008804051 0 0) 0x7f674c09dd10 con 0x7f674c13c220 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient(hunting): _negotiate 2024-09-15T20:44:56.779 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient(hunting): my global_id is 124274 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6738004ef0 con 0x7f674c13c220 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 client.?.objecter ms_handle_connect 0x7f674c0a24e0 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2821412614 0 0) 0x7f6738004ef0 con 0x7f674c13c220 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f674c09dd10 con 0x7f674c13c220 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f673c0040c0 con 0x7f674c13c220 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:44:56.780 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient(hunting): dump: 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:44:56.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 252539815 0 0) 0x7f674c09dd10 con 0x7f674c13c220 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 >> v1:172.21.15.146:6789/0 conn(0x7f674c0a24e0 legacy=0x7f674c071a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 >> v1:172.21.15.78:6789/0 conn(0x7f674c135b60 legacy=0x7f674c0721a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient: _finish_hunting 0 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 monclient: found mon.c 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f674c09dff0 con 0x7f674c13c220 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient: _finish_auth 0 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 10 monclient: _check_auth_tickets 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f674affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:44:56.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 5 monclient: authenticate success, global_id 124274 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: _renew_subs 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f674c13af70 con 0x7f674c13c220 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 client.124274.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: _renew_subs 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f674c0736b0 con 0x7f674c13c220 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.762+0000 7f6754304640 1 librados: init done 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f673c0042e0 con 0x7f674c13c220 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f673c0046f0 con 0x7f674c13c220 2024-09-15T20:44:56.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f675089d640 4 set_mon_vals no callback set 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 10 monclient: dump: 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:44:56.784 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f6754304640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f6754304640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f6754304640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f672c005090 con 0x7f674c13c220 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f673c004890 con 0x7f674c13c220 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7f673c04b890 con 0x7f674c13c220 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 10 client.124274.objecter ms_dispatch 0x7f674c000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 3 client.124274.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 3 client.124274.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 20 client.124274.objecter dump_active .. 0 homeless 2024-09-15T20:44:56.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f67520a0640 10 monclient: get_auth_request con 0x7f673803df10 auth_method 0 2024-09-15T20:44:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.766+0000 7f674affd640 10 client.124274.objecter ms_handle_connect 0x7f673803df10 2024-09-15T20:44:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.770+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f673c016910 con 0x7f674c13c220 2024-09-15T20:44:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.770+0000 7f674affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:44:56.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.770+0000 7f674affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:44:56.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.986+0000 7f6754304640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:44:56.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.986+0000 7f6754304640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:44:56.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.986+0000 7f6754304640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:44:56.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.986+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f672c005380 con 0x7f674c13c220 2024-09-15T20:44:56.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.986+0000 7f674affd640 1 -- 172.21.15.78:0/439057552 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 1625194501) 0x7f673c01acf0 con 0x7f674c13c220 2024-09-15T20:44:56.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.986+0000 7f674affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:44:56.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.986+0000 7f674affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:44:56.990 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":54,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 10 librados: watch_flush enter 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 10 librados: watch_flush exit 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 20 client.124274.objecter shutdown clearing up homeless session... 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 10 client.124274.objecter successfully canceled tick 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 >> v1:172.21.15.146:6812/435598642 conn(0x7f673803df10 legacy=0x7f67380403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 10 monclient: shutdown 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 >> v1:172.21.15.78:6790/0 conn(0x7f674c13c220 legacy=0x7f674c1d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 shutdown_connections 2024-09-15T20:44:56.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 >> 172.21.15.78:0/439057552 conn(0x7f674c0a63c0 msgr2=0x7f674c12ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:44:56.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 shutdown_connections 2024-09-15T20:44:56.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 1 -- 172.21.15.78:0/439057552 wait complete. 2024-09-15T20:44:56.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:44:56.990+0000 7f6754304640 1 librados: shutdown 2024-09-15T20:45:00.009 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient: get_monmap_and_config 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient: build_initial_monmap 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient: monmap: 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:45:00.134729+0000 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:45:00.134729+0000 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:45:00.133 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:00.134 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:45:00.134 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:45:00.134 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:45:00.134 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:00.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 1 Processor -- start 2024-09-15T20:45:00.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 1 -- start start 2024-09-15T20:45:00.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient: init 2024-09-15T20:45:00.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient: _reopen_session rank -1 2024-09-15T20:45:00.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient(hunting): picked mon.noname-b con 0x7fd3a8137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient(hunting): picked mon.noname-c con 0x7fd3a8074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient(hunting): picked mon.noname-a con 0x7fd3a813e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a8138850 con 0x7fd3a813e370 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a80a5da0 con 0x7fd3a8074500 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a8073560 con 0x7fd3a8137d60 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient(hunting): _renew_subs 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3b0779640 10 monclient(hunting): authenticate will time out at 1961.019338s 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3aed16640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd3a8140760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3ae515640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd3a8137d60 0x7fd3a8138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50708/0 (socket says 172.21.15.78:50708) 2024-09-15T20:45:00.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.130+0000 7fd3ae515640 1 -- 172.21.15.78:0/2799534115 learned_addr learned my addr 172.21.15.78:0/2799534115 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:45:00.336 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.334+0000 7fd3aed16640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd3a8140760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:00.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:00.738+0000 7fd3aed16640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd3a8140760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:01.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:01.542+0000 7fd3aed16640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd3a8140760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 10 monclient(hunting): tick 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 1 monclient(hunting): continuing hunt 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6790/0 conn(0x7fd3a8137d60 legacy=0x7fd3a8138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.146:6789/0 conn(0x7fd3a8074500 legacy=0x7fd3a813cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd3a8140760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:45:03.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c0036d0 addr v1:172.21.15.146:6789/0 2024-09-15T20:45:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fd3a813e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:45:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fd3a8137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:45:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a8073840 con 0x7fd3a813e370 2024-09-15T20:45:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a8138850 con 0x7fd38c0036d0 2024-09-15T20:45:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd38c0103d0 con 0x7fd3a8137d60 2024-09-15T20:45:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd39ffff640 10 monclient(hunting): _renew_subs 2024-09-15T20:45:03.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.130+0000 7fd3add14640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c0093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:03.335 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.334+0000 7fd3add14640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c0093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:03.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:03.738+0000 7fd3add14640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c0093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:04.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:04.542+0000 7fd3add14640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c0093d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:06.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.130+0000 7fd39ffff640 10 monclient(hunting): tick 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.130+0000 7fd39ffff640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.130+0000 7fd39ffff640 1 monclient(hunting): continuing hunt 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.130+0000 7fd39ffff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.130+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6790/0 conn(0x7fd3a8137d60 legacy=0x7fd38c00cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.130+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.146:6789/0 conn(0x7fd38c0036d0 legacy=0x7fd38c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c0093d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c00f400 addr v1:172.21.15.146:6789/0 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 10 monclient(hunting): picked mon.noname-b con 0x7fd3a8137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 10 monclient(hunting): picked mon.noname-a con 0x7fd3a813e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd38c011730 con 0x7fd3a813e370 2024-09-15T20:45:06.137 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a8073840 con 0x7fd38c00f400 2024-09-15T20:45:06.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 1 -- 172.21.15.78:0/2799534115 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd38c010100 con 0x7fd3a8137d60 2024-09-15T20:45:06.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd39ffff640 10 monclient(hunting): _renew_subs 2024-09-15T20:45:06.138 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.134+0000 7fd3aed16640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:06.339 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.338+0000 7fd3aed16640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:06.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:06.742+0000 7fd3aed16640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c005ac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:07.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2085789640 0 0) 0x7fd38c010100 con 0x7fd3a8137d60 2024-09-15T20:45:07.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient(hunting): my global_id is 124277 2024-09-15T20:45:07.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd394003530 con 0x7fd3a8137d60 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3149737914 0 0) 0x7fd394003530 con 0x7fd3a8137d60 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.146:6789/0 conn(0x7fd38c00f400 legacy=0x7fd38c00f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6789/0 conn(0x7fd3a813e370 legacy=0x7fd38c005ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: _finish_hunting 0 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 monclient: found mon.noname-b 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd38c00cc90 con 0x7fd3a8137d60 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: _finish_auth 0 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: _check_auth_tickets 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd3a4001660 con 0x7fd3a8137d60 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3b0779640 5 monclient: authenticate success, global_id 124277 2024-09-15T20:45:07.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3b0779640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: dump: 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.392 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3b0779640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd3a4001800 con 0x7fd3a8137d60 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: handle_config config(0 keys) 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 1 -- 172.21.15.78:0/2799534115 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd3a4001c50 con 0x7fd3a8137d60 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3b0779640 10 monclient: get_monmap_and_config success 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3b0779640 4 set_mon_vals no callback set 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.386+0000 7fd3ad513640 10 monclient: dump: 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.393 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.394 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.394 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.394 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.394 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.394 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.394 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 10 monclient: shutdown 2024-09-15T20:45:07.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:45:07.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 -- 172.21.15.78:0/2799534115 >> v1:172.21.15.78:6790/0 conn(0x7fd3a8137d60 legacy=0x7fd38c0062f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 -- 172.21.15.78:0/2799534115 shutdown_connections 2024-09-15T20:45:07.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 -- 172.21.15.78:0/2799534115 >> 172.21.15.78:0/2799534115 conn(0x7fd3a80aa650 msgr2=0x7fd3a80aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 -- 172.21.15.78:0/2799534115 shutdown_connections 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 -- 172.21.15.78:0/2799534115 wait complete. 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 10 monclient: build_initial_monmap 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 10 monclient: monmap: 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:45:07.399 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 librados: starting msgr at 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 librados: starting objecter 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 Processor -- start 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 -- start start 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 librados: setting wanted keys 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 1 librados: calling monclient init 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.390+0000 7fd3b0779640 10 monclient: init 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient: _reopen_session rank -1 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient(hunting): picked mon.noname-c con 0x7fd38c00f400 addr v1:172.21.15.78:6790/0 2024-09-15T20:45:07.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient(hunting): picked mon.noname-a con 0x7fd3a8137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient(hunting): picked mon.noname-b con 0x7fd3a813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a809e0b0 con 0x7fd3a8137d60 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a813d540 con 0x7fd3a813e370 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3a813d820 con 0x7fd38c00f400 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient(hunting): _renew_subs 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient(hunting): authenticate will time out at 1968.280201s 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3add14640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fd3a8137d60 legacy=0x7fd3a81d1ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3ae515640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fd38c00f400 0x7fd3a8134e70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50738/0 (socket says 172.21.15.78:50738) 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3ae515640 1 -- 172.21.15.78:0/2968010726 learned_addr learned my addr 172.21.15.78:0/2968010726 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 client.?.objecter ms_handle_connect 0x7fd38c00f400 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:45:07.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1370073432 0 0) 0x7fd3a813d820 con 0x7fd38c00f400 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient(hunting): my global_id is 124280 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd38c010100 con 0x7fd38c00f400 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 client.?.objecter ms_handle_connect 0x7fd3a813e370 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2084313658 0 0) 0x7fd38c010100 con 0x7fd38c00f400 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fd3a813d820 con 0x7fd38c00f400 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd3980016d0 con 0x7fd38c00f400 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient(hunting): dump: 2024-09-15T20:45:07.402 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2382998805 0 0) 0x7fd3a813d820 con 0x7fd38c00f400 2024-09-15T20:45:07.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 >> v1:172.21.15.146:6789/0 conn(0x7fd3a813e370 legacy=0x7fd3a81d53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 >> v1:172.21.15.78:6789/0 conn(0x7fd3a8137d60 legacy=0x7fd3a81d1ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient: _finish_hunting 0 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 monclient: found mon.c 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3a80a9810 con 0x7fd38c00f400 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient: _finish_auth 0 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient: _check_auth_tickets 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 5 monclient: authenticate success, global_id 124280 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient: _renew_subs 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fd3a813e050 con 0x7fd38c00f400 2024-09-15T20:45:07.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 client.124280.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient: _renew_subs 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fd3a8135580 con 0x7fd38c00f400 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 1 librados: init done 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fd398001c60 con 0x7fd38c00f400 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fd398004990 con 0x7fd38c00f400 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 monclient: dump: 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.405 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3acd12640 4 set_mon_vals no callback set 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fd398004bb0 con 0x7fd38c00f400 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd390005090 con 0x7fd38c00f400 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7fd39804b8b0 con 0x7fd38c00f400 2024-09-15T20:45:07.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 10 client.124280.objecter ms_dispatch 0x7fd3a8000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:45:07.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 3 client.124280.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:45:07.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 3 client.124280.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:45:07.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd39effd640 20 client.124280.objecter dump_active .. 0 homeless 2024-09-15T20:45:07.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.394+0000 7fd3add14640 10 monclient: get_auth_request con 0x7fd38c03eea0 auth_method 0 2024-09-15T20:45:07.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.398+0000 7fd39effd640 10 client.124280.objecter ms_handle_connect 0x7fd38c03eea0 2024-09-15T20:45:07.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.398+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fd398016a50 con 0x7fd38c00f400 2024-09-15T20:45:07.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.398+0000 7fd39effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:45:07.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.398+0000 7fd39effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:45:07.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.642+0000 7fd3b0779640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:45:07.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.642+0000 7fd3b0779640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:45:07.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.642+0000 7fd3b0779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.642+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd390005380 con 0x7fd38c00f400 2024-09-15T20:45:07.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.642+0000 7fd39effd640 1 -- 172.21.15.78:0/2968010726 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 572804760) 0x7fd398016350 con 0x7fd38c00f400 2024-09-15T20:45:07.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.642+0000 7fd39effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:45:07.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.642+0000 7fd39effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:45:07.646 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":56,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:45:07.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 10 librados: watch_flush enter 2024-09-15T20:45:07.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 10 librados: watch_flush exit 2024-09-15T20:45:07.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 20 client.124280.objecter shutdown clearing up homeless session... 2024-09-15T20:45:07.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 10 client.124280.objecter successfully canceled tick 2024-09-15T20:45:07.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 >> v1:172.21.15.146:6812/435598642 conn(0x7fd38c03eea0 legacy=0x7fd38c041360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 10 monclient: shutdown 2024-09-15T20:45:07.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:45:07.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 >> v1:172.21.15.78:6790/0 conn(0x7fd38c00f400 legacy=0x7fd3a8134e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 shutdown_connections 2024-09-15T20:45:07.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 >> 172.21.15.78:0/2968010726 conn(0x7fd3a80aa650 msgr2=0x7fd3a80a8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:45:07.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 shutdown_connections 2024-09-15T20:45:07.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 1 -- 172.21.15.78:0/2968010726 wait complete. 2024-09-15T20:45:07.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.646+0000 7fd3b0779640 1 librados: shutdown 2024-09-15T20:45:07.664 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:45:07.665 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T20:45:07.665 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:45:07.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.782+0000 7fb2d98af640 10 monclient: get_monmap_and_config 2024-09-15T20:45:07.785 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.782+0000 7fb2d98af640 10 monclient: build_initial_monmap 2024-09-15T20:45:07.786 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.782+0000 7fb2d98af640 10 monclient: monmap: 2024-09-15T20:45:07.786 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:45:07.786 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.786 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:45:07.787407+0000 2024-09-15T20:45:07.786 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:45:07.787407+0000 2024-09-15T20:45:07.786 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:45:07.786 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.787 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:45:07.787 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:45:07.787 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:45:07.787 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.782+0000 7fb2d98af640 1 Processor -- start 2024-09-15T20:45:07.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.782+0000 7fb2d98af640 1 -- start start 2024-09-15T20:45:07.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.782+0000 7fb2d98af640 10 monclient: init 2024-09-15T20:45:07.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: _reopen_session rank -1 2024-09-15T20:45:07.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:45:07.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d4137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:45:07.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d4074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:45:07.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d413e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:45:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d4138850 con 0x7fb2d413e370 2024-09-15T20:45:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d40a5da0 con 0x7fb2d4137d60 2024-09-15T20:45:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d4073560 con 0x7fb2d4074500 2024-09-15T20:45:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): _renew_subs 2024-09-15T20:45:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): authenticate will time out at 1968.672811s 2024-09-15T20:45:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d37fe640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fb2d413e370 legacy=0x7fb2d4140760 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:07.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d27fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb2d4074500 0x7fb2d413cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50754/0 (socket says 172.21.15.78:50754) 2024-09-15T20:45:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d27fc640 1 -- 172.21.15.78:0/3814296647 learned_addr learned my addr 172.21.15.78:0/3814296647 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:45:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246420594 0 0) 0x7fb2d4073560 con 0x7fb2d4074500 2024-09-15T20:45:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient(hunting): my global_id is 124283 2024-09-15T20:45:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2bc003530 con 0x7fb2d4074500 2024-09-15T20:45:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1633514118 0 0) 0x7fb2bc003530 con 0x7fb2d4074500 2024-09-15T20:45:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 >> v1:172.21.15.146:6789/0 conn(0x7fb2d4137d60 legacy=0x7fb2d4138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 >> v1:172.21.15.78:6789/0 conn(0x7fb2d413e370 legacy=0x7fb2d4140760 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:45:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:45:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 monclient: found mon.noname-b 2024-09-15T20:45:07.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:45:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d4073840 con 0x7fb2d4074500 2024-09-15T20:45:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: _finish_auth 0 2024-09-15T20:45:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:45:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:45:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2c80016a0 con 0x7fb2d4074500 2024-09-15T20:45:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:45:07.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: dump: 2024-09-15T20:45:07.810 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.810 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.810 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.810 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.810 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.810 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.811 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.811 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.811 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.811 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 5 monclient: authenticate success, global_id 124283 2024-09-15T20:45:07.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:45:07.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb2c80018c0 con 0x7fb2d4074500 2024-09-15T20:45:07.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:45:07.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: get_monmap_and_config success 2024-09-15T20:45:07.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 4 set_mon_vals no callback set 2024-09-15T20:45:07.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 1 -- 172.21.15.78:0/3814296647 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2c8001cd0 con 0x7fb2d4074500 2024-09-15T20:45:07.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:45:07.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d1ffb640 10 monclient: dump: 2024-09-15T20:45:07.812 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.813 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.814 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: shutdown 2024-09-15T20:45:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:45:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- 172.21.15.78:0/3814296647 >> v1:172.21.15.78:6790/0 conn(0x7fb2d4074500 legacy=0x7fb2d413cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- 172.21.15.78:0/3814296647 shutdown_connections 2024-09-15T20:45:07.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- 172.21.15.78:0/3814296647 >> 172.21.15.78:0/3814296647 conn(0x7fb2d40aa650 msgr2=0x7fb2d40aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:45:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- 172.21.15.78:0/3814296647 shutdown_connections 2024-09-15T20:45:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- 172.21.15.78:0/3814296647 wait complete. 2024-09-15T20:45:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: build_initial_monmap 2024-09-15T20:45:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: monmap: 2024-09-15T20:45:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:45:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:45:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:45:07.815 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:45:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:45:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:45:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:45:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:45:07.816 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 librados: starting msgr at 2024-09-15T20:45:07.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 librados: starting objecter 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 Processor -- start 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- start start 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 librados: setting wanted keys 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 librados: calling monclient init 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: init 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: _reopen_session rank -1 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): picked mon.noname-b con 0x7fb2d4074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): picked mon.noname-a con 0x7fb2d4137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:45:07.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): picked mon.noname-c con 0x7fb2d413e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d413c6a0 con 0x7fb2d4137d60 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d409e0b0 con 0x7fb2d4074500 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2d413a340 con 0x7fb2d413e370 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): _renew_subs 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d98af640 10 monclient(hunting): authenticate will time out at 1968.675647s 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d27fc640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fb2d4137d60 legacy=0x7fb2d4138f90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d37fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb2d413e370 0x7fb2d41396a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50766/0 (socket says 172.21.15.78:50766) 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2d37fe640 1 -- 172.21.15.78:0/3895932128 learned_addr learned my addr 172.21.15.78:0/3895932128 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2bbfff640 10 client.?.objecter ms_handle_connect 0x7fb2d413e370 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2bbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3469971194 0 0) 0x7fb2d413a340 con 0x7fb2d413e370 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2bbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.786+0000 7fb2bbfff640 10 monclient(hunting): my global_id is 124286 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2c0004ec0 con 0x7fb2d413e370 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 client.?.objecter ms_handle_connect 0x7fb2d4074500 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2514600082 0 0) 0x7fb2c0004ec0 con 0x7fb2d413e370 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2d413a340 con 0x7fb2d413e370 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2cc004440 con 0x7fb2d413e370 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): dump: 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.819 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3522967680 0 0) 0x7fb2d409e0b0 con 0x7fb2d4074500 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): my global_id is 134161 2024-09-15T20:45:07.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2c0004ec0 con 0x7fb2d4074500 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1218113250 0 0) 0x7fb2d413a340 con 0x7fb2d413e370 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 >> v1:172.21.15.146:6789/0 conn(0x7fb2d4074500 legacy=0x7fb2d4138880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 >> v1:172.21.15.78:6789/0 conn(0x7fb2d4137d60 legacy=0x7fb2d4138f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: _finish_hunting 0 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 monclient: found mon.c 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d413be00 con 0x7fb2d413e370 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: _finish_auth 0 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: _check_auth_tickets 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 5 monclient: authenticate success, global_id 124286 2024-09-15T20:45:07.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 10 monclient: _renew_subs 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d413c0e0 con 0x7fb2d413e370 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 10 client.124286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 10 monclient: _renew_subs 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb2d41d5e10 con 0x7fb2d413e370 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 1 librados: init done 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb2cc004660 con 0x7fb2d413e370 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2cc004a70 con 0x7fb2d413e370 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d17fa640 4 set_mon_vals no callback set 2024-09-15T20:45:07.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: dump: 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb2b40065d0 con 0x7fb2b4002870 2024-09-15T20:45:07.823 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 client.124286.objecter ms_handle_connect 0x7fb2b4002870 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 client.124286.objecter resend_mon_ops 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb2cc004c10 con 0x7fb2d413e370 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 827779702 0 0) 0x7fb2b40065d0 con 0x7fb2b4002870 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient(hunting): my global_id is 124289 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2d413a340 con 0x7fb2b4002870 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7fb2cc04bad0 con 0x7fb2d413e370 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 client.124286.objecter ms_dispatch 0x7fb2d4000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 3 client.124286.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 3 client.124286.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:45:07.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 20 client.124286.objecter dump_active .. 0 homeless 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 901701833 0 0) 0x7fb2d413a340 con 0x7fb2b4002870 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2b40065d0 con 0x7fb2b4002870 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2c40017a0 con 0x7fb2b4002870 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: dump: 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.825 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2d27fc640 10 monclient: get_auth_request con 0x7fb2c003e000 auth_method 0 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 927876008 0 0) 0x7fb2b40065d0 con 0x7fb2b4002870 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2b400a170 con 0x7fb2b4002870 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb2c4001ae0 con 0x7fb2b4002870 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 >> v1:172.21.15.78:6790/0 conn(0x7fb2b4002870 legacy=0x7fb2b4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 >> v1:172.21.15.78:6790/0 conn(0x7fb2b4002870 legacy=0x7fb2b4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.790+0000 7fb2bbfff640 10 client.124286.objecter ms_handle_connect 0x7fb2c003e000 2024-09-15T20:45:07.826 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb2b4009e80 con 0x7fb2b4009140 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 client.124286.objecter ms_handle_connect 0x7fb2b4009140 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 client.124286.objecter resend_mon_ops 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2850092885 0 0) 0x7fb2b4009e80 con 0x7fb2b4009140 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient(hunting): my global_id is 124292 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb2b40065d0 con 0x7fb2b4009140 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3568679587 0 0) 0x7fb2b40065d0 con 0x7fb2b4009140 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb2b4009e80 con 0x7fb2b4009140 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb2c40037f0 con 0x7fb2b4009140 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient: dump: 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1995826675 0 0) 0x7fb2b4009e80 con 0x7fb2b4009140 2024-09-15T20:45:07.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb2b400c420 con 0x7fb2b4009140 2024-09-15T20:45:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4294 (unknown 0 0 1054543410) 0x7fb2c4005a30 con 0x7fb2b4009140 2024-09-15T20:45:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:45:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:45:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 >> v1:172.21.15.78:6790/0 conn(0x7fb2b4009140 legacy=0x7fb2b40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2bbfff640 1 -- 172.21.15.78:0/3895932128 >> v1:172.21.15.78:6790/0 conn(0x7fb2b4009140 legacy=0x7fb2b40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:45:07.829 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:45:07.829 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:45:07.829 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 56, 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 125726, 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:45:07.830 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:45:07.831 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.832 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:45:07.833 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:45:07.834 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:45:07.835 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.836 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.837 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:45:07.838 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:45:07.839 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:07.840 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 7 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:45:07.841 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:07.842 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:07.842 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:07.842 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:45:07.842 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:45:07.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 10 librados: watch_flush enter 2024-09-15T20:45:07.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 10 librados: watch_flush exit 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 20 client.124286.objecter shutdown clearing up homeless session... 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 10 client.124286.objecter successfully canceled tick 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 >> v1:172.21.15.146:6812/435598642 conn(0x7fb2c003e000 legacy=0x7fb2c00404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 10 monclient: shutdown 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 >> v1:172.21.15.78:6790/0 conn(0x7fb2d413e370 legacy=0x7fb2d41396a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d37fe640 1 -- 172.21.15.78:0/3895932128 reap_dead start 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d37fe640 1 -- 172.21.15.78:0/3895932128 reap_dead start 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 shutdown_connections 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 >> 172.21.15.78:0/3895932128 conn(0x7fb2d40aa650 msgr2=0x7fb2d40a8970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 shutdown_connections 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 1 -- 172.21.15.78:0/3895932128 wait complete. 2024-09-15T20:45:07.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.798+0000 7fb2d98af640 1 librados: shutdown 2024-09-15T20:45:07.845 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:45:07.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T20:45:07.845 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:45:07.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient: get_monmap_and_config 2024-09-15T20:45:07.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient: build_initial_monmap 2024-09-15T20:45:07.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient: monmap: 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:45:07.969476+0000 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:45:07.969476+0000 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:45:07.968 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 Processor -- start 2024-09-15T20:45:07.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- start start 2024-09-15T20:45:07.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient: init 2024-09-15T20:45:07.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient: _reopen_session rank -1 2024-09-15T20:45:07.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:45:07.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient(hunting): picked mon.noname-c con 0x7fc1f0135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:45:07.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient(hunting): picked mon.noname-a con 0x7fc1f00a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:45:07.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient(hunting): picked mon.noname-b con 0x7fc1f013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:45:07.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f00745e0 con 0x7fc1f00a24e0 2024-09-15T20:45:07.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f006ecf0 con 0x7fc1f0135b60 2024-09-15T20:45:07.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f006efd0 con 0x7fc1f013c220 2024-09-15T20:45:07.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient(hunting): _renew_subs 2024-09-15T20:45:07.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1ee7fc640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fc1f00a24e0 legacy=0x7fc1f013ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:07.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient(hunting): authenticate will time out at 1968.854143s 2024-09-15T20:45:07.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1ef7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc1f013c220 0x7fc1f013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50808/0 (socket says 172.21.15.78:50808) 2024-09-15T20:45:07.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1ef7fe640 1 -- 172.21.15.78:0/2294275783 learned_addr learned my addr 172.21.15.78:0/2294275783 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:45:07.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3006540948 0 0) 0x7fc1f006efd0 con 0x7fc1f013c220 2024-09-15T20:45:07.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient(hunting): my global_id is 124295 2024-09-15T20:45:07.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1dc003530 con 0x7fc1f013c220 2024-09-15T20:45:07.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3408698440 0 0) 0x7fc1dc003530 con 0x7fc1f013c220 2024-09-15T20:45:07.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 >> v1:172.21.15.146:6789/0 conn(0x7fc1f0135b60 legacy=0x7fc1f0135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 >> v1:172.21.15.78:6789/0 conn(0x7fc1f00a24e0 legacy=0x7fc1f013ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:45:07.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: _finish_hunting 0 2024-09-15T20:45:07.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 monclient: found mon.noname-b 2024-09-15T20:45:07.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f0136650 con 0x7fc1f013c220 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: _finish_auth 0 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: _check_auth_tickets 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 5 monclient: authenticate success, global_id 124295 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc1e80016a0 con 0x7fc1f013c220 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: dump: 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.985 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1e80018c0 con 0x7fc1f013c220 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:45:07.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 1 -- 172.21.15.78:0/2294275783 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc1e8001cd0 con 0x7fc1f013c220 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient: get_monmap_and_config success 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 4 set_mon_vals no callback set 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1edffb640 10 monclient: dump: 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.987 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 10 monclient: shutdown 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- 172.21.15.78:0/2294275783 >> v1:172.21.15.78:6790/0 conn(0x7fc1f013c220 legacy=0x7fc1f013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- 172.21.15.78:0/2294275783 shutdown_connections 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- 172.21.15.78:0/2294275783 >> 172.21.15.78:0/2294275783 conn(0x7fc1f00a63c0 msgr2=0x7fc1f00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- 172.21.15.78:0/2294275783 shutdown_connections 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.966+0000 7fc1f55b4640 1 -- 172.21.15.78:0/2294275783 wait complete. 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: build_initial_monmap 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: monmap: 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:45:07.988 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 librados: starting msgr at 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 librados: starting objecter 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 Processor -- start 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 -- start start 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 librados: setting wanted keys 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 librados: calling monclient init 2024-09-15T20:45:07.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: init 2024-09-15T20:45:07.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: _reopen_session rank -1 2024-09-15T20:45:07.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:45:07.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient(hunting): picked mon.noname-a con 0x7fc1f00a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:45:07.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient(hunting): picked mon.noname-b con 0x7fc1f0135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:45:07.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient(hunting): picked mon.noname-c con 0x7fc1f013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:45:07.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f00728b0 con 0x7fc1f00a24e0 2024-09-15T20:45:07.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1eeffd640 1 -- >> v1:172.21.15.78:6789/0 conn(0x7fc1f00a24e0 legacy=0x7fc1f0071a90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6789/0 2024-09-15T20:45:07.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f009d180 con 0x7fc1f0135b60 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1f013fa50 con 0x7fc1f013c220 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient(hunting): _renew_subs 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient(hunting): authenticate will time out at 1968.857208s 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1ef7fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc1f013c220 0x7fc1f01d4c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:50822/0 (socket says 172.21.15.78:50822) 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1ef7fe640 1 -- 172.21.15.78:0/4190086720 learned_addr learned my addr 172.21.15.78:0/4190086720 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 client.?.objecter ms_handle_connect 0x7fc1f013c220 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3536153439 0 0) 0x7fc1f013fa50 con 0x7fc1f013c220 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:07.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): my global_id is 124298 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d0004ec0 con 0x7fc1f013c220 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 client.?.objecter ms_handle_connect 0x7fc1f0135b60 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 982815696 0 0) 0x7fc1d0004ec0 con 0x7fc1f013c220 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1f013fa50 con 0x7fc1f013c220 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc1e8001c10 con 0x7fc1f013c220 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:45:07.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): dump: 2024-09-15T20:45:07.993 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.993 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.993 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.993 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.993 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.993 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.993 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.994 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.994 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.994 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:45:07.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1411193654 0 0) 0x7fc1f013fa50 con 0x7fc1f013c220 2024-09-15T20:45:07.994 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 >> v1:172.21.15.146:6789/0 conn(0x7fc1f0135b60 legacy=0x7fc1f00721a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:07.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 >> v1:172.21.15.78:6789/0 conn(0x7fc1f00a24e0 legacy=0x7fc1f0071a90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:45:07.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:45:07.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 monclient: found mon.c 2024-09-15T20:45:07.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:45:07.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1f009d490 con 0x7fc1f013c220 2024-09-15T20:45:07.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: _finish_auth 0 2024-09-15T20:45:07.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:45:07.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:45:07.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 5 monclient: authenticate success, global_id 124298 2024-09-15T20:45:07.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: _renew_subs 2024-09-15T20:45:07.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc1f009dd10 con 0x7fc1f013c220 2024-09-15T20:45:07.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 client.124298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:45:07.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: _renew_subs 2024-09-15T20:45:07.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:45:07.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fc1f00734c0 con 0x7fc1f013c220 2024-09-15T20:45:07.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 librados: init done 2024-09-15T20:45:07.997 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc1e80011b0 con 0x7fc1f013c220 2024-09-15T20:45:07.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:45:07.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc1e8004c00 con 0x7fc1f013c220 2024-09-15T20:45:07.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:07.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:45:07.998 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: dump: 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1ed7fa640 4 set_mon_vals no callback set 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:45:07.999 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc1d40065d0 con 0x7fc1d4002870 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fc1e8010c70 con 0x7fc1f013c220 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(535..535 src has 379..535) ==== 5801+0+0 (unknown 2071711608 0 0) 0x7fc1e804b960 con 0x7fc1f013c220 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 client.124298.objecter ms_dispatch 0x7fc1f0000ba0 osd_map(535..535 src has 379..535) 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 3 client.124298.objecter handle_osd_map got epochs [535,535] > 0 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 3 client.124298.objecter handle_osd_map decoding full epoch 535 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 20 client.124298.objecter dump_active .. 0 homeless 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 client.124298.objecter ms_handle_connect 0x7fc1d4002870 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 client.124298.objecter resend_mon_ops 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1ee7fc640 10 monclient: get_auth_request con 0x7fc1d003deb0 auth_method 0 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3375518653 0 0) 0x7fc1d40065d0 con 0x7fc1d4002870 2024-09-15T20:45:08.000 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient(hunting): my global_id is 134170 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1f013fa50 con 0x7fc1d4002870 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 905103004 0 0) 0x7fc1f013fa50 con 0x7fc1d4002870 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1d40065d0 con 0x7fc1d4002870 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 client.124298.objecter ms_handle_connect 0x7fc1d003deb0 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc1e0001840 con 0x7fc1d4002870 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:08.001 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.970+0000 7fc1e77fe640 10 monclient: dump: 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.974+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4016498208 0 0) 0x7fc1d40065d0 con 0x7fc1d4002870 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.974+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc1d400a170 con 0x7fc1d4002870 2024-09-15T20:45:08.002 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.974+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc1e0001b20 con 0x7fc1d4002870 2024-09-15T20:45:08.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.974+0000 7fc1e77fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:45:08.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.974+0000 7fc1e77fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:45:08.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.974+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 >> v1:172.21.15.146:6789/0 conn(0x7fc1d4002870 legacy=0x7fc1d4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:08.003 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.974+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 >> v1:172.21.15.146:6789/0 conn(0x7fc1d4002870 legacy=0x7fc1d4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:45:08.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1f55b4640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc1d4009e80 con 0x7fc1d4009140 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1e77fe640 10 client.124298.objecter ms_handle_connect 0x7fc1d4009140 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1e77fe640 10 client.124298.objecter resend_mon_ops 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2293393970 0 0) 0x7fc1d4009e80 con 0x7fc1d4009140 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1e77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1e77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1e77fe640 10 monclient(hunting): my global_id is 134173 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.978+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1d40065d0 con 0x7fc1d4009140 2024-09-15T20:45:08.005 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 632973567 0 0) 0x7fc1d40065d0 con 0x7fc1d4009140 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc1d4009e80 con 0x7fc1d4009140 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc1e00037a0 con 0x7fc1d4009140 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 10 monclient: dump: 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:45:08.006 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2819717212 0 0) 0x7fc1d4009e80 con 0x7fc1d4009140 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc1d400c420 con 0x7fc1d4009140 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3961 (unknown 0 0 2044550365) 0x7fc1e00057d0 con 0x7fc1d4009140 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 >> v1:172.21.15.146:6789/0 conn(0x7fc1d4009140 legacy=0x7fc1d40095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1e77fe640 1 -- 172.21.15.78:0/4190086720 >> v1:172.21.15.146:6789/0 conn(0x7fc1d4009140 legacy=0x7fc1d40095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:45:08.007 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 56, 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17851, 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:45:08.008 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:08.009 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:45:08.010 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:45:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:45:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:45:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:45:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:45:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:45:08.011 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:45:08.012 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:45:08.013 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:08.014 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:45:08.015 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:45:08.016 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:45:08.017 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:45:08.018 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:45:08.019 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:45:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:45:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:45:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:45:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:45:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:45:08.020 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:45:08.020 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:45:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 10 librados: watch_flush enter 2024-09-15T20:45:08.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 10 librados: watch_flush exit 2024-09-15T20:45:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 20 client.124298.objecter shutdown clearing up homeless session... 2024-09-15T20:45:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 10 client.124298.objecter successfully canceled tick 2024-09-15T20:45:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 >> v1:172.21.15.146:6812/435598642 conn(0x7fc1d003deb0 legacy=0x7fc1d0040370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 10 monclient: shutdown 2024-09-15T20:45:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:45:08.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 >> v1:172.21.15.78:6790/0 conn(0x7fc1f013c220 legacy=0x7fc1f01d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:45:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1ef7fe640 1 -- 172.21.15.78:0/4190086720 reap_dead start 2024-09-15T20:45:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1ef7fe640 1 -- 172.21.15.78:0/4190086720 reap_dead start 2024-09-15T20:45:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 shutdown_connections 2024-09-15T20:45:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 >> 172.21.15.78:0/4190086720 conn(0x7fc1f00a63c0 msgr2=0x7fc1f012e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:45:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 shutdown_connections 2024-09-15T20:45:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 1 -- 172.21.15.78:0/4190086720 wait complete. 2024-09-15T20:45:08.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:45:07.982+0000 7fc1f55b4640 1 librados: shutdown 2024-09-15T20:45:08.025 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:45:08.026 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:45:55.169 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:45:55.169 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringFlush (8993 ms) 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestSnapHasChunk 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestSnapHasChunk (5968 ms) 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollback 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollback (5065 ms) 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestRollbackRefcount 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestRollbackRefcount (25968 ms) 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.ManifestEvictRollback 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.ManifestEvictRollback (14077 ms) 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.PropagateBaseTierError 2024-09-15T20:45:55.170 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.PropagateBaseTierError (26911 ms) 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.HelloWriteReturn 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: 00000000 79 6f 75 20 6d 69 67 68 74 20 73 65 65 20 74 68 |you might see th| 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: 00000010 69 73 |is| 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: 00000012 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.HelloWriteReturn (12098 ms) 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: require_osd_release = squid 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsPP.TierFlushDuringUnsetDedupTier (5909 ms) 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] 48 tests from LibRadosTwoPoolsPP (640707 ms total) 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.Dirty 2024-09-15T20:45:55.171 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.Dirty (1035 ms) 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.FlushWriteRaces 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.FlushWriteRaces (11018 ms) 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.CallForcesPromote 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.CallForcesPromote (17979 ms) 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTierECPP.HitSetNone 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTierECPP.HitSetNone (4 ms) 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] 4 tests from LibRadosTierECPP (30036 ms total) 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Overlay 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Overlay (7074 ms) 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Promote 2024-09-15T20:45:55.172 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Promote (7974 ms) 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnap 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: waiting for scrub... 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: done waiting 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnap (32207 ms) 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteSnapTrimRace (9692 ms) 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Whiteout 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Whiteout (8071 ms) 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Evict 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Evict (7915 ms) 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.EvictSnap 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.EvictSnap (10081 ms) 2024-09-15T20:45:55.173 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlush 2024-09-15T20:46:38.027 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-09-15T20:46:38.027 INFO:tasks.ceph.mon.a:Restarting daemon 2024-09-15T20:46:38.027 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-09-15T20:46:38.031 INFO:tasks.ceph.mon.a:Started 2024-09-15T20:46:38.031 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:46:38.031 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:46:38.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient: get_monmap_and_config 2024-09-15T20:46:38.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient: build_initial_monmap 2024-09-15T20:46:38.163 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient: monmap: 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:38.165867+0000 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:38.165867+0000 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:38.164 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 1 Processor -- start 2024-09-15T20:46:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 1 -- start start 2024-09-15T20:46:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient: init 2024-09-15T20:46:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.167 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient(hunting): picked mon.noname-a con 0x7fba24137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient(hunting): picked mon.noname-c con 0x7fba24074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient(hunting): picked mon.noname-b con 0x7fba2413e370 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba24138850 con 0x7fba24137d60 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba240a5da0 con 0x7fba24074500 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba24073560 con 0x7fba2413e370 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient(hunting): authenticate will time out at 2059.050352s 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba22ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fba24137d60 0x7fba24138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34614/0 (socket says 172.21.15.78:34614) 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba22ffd640 1 -- 172.21.15.78:0/979645153 learned_addr learned my addr 172.21.15.78:0/979645153 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3309694412 0 0) 0x7fba24073560 con 0x7fba2413e370 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.168 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient(hunting): my global_id is 124301 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba0c003530 con 0x7fba2413e370 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3513609341 0 0) 0x7fba0c003530 con 0x7fba2413e370 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 >> v1:172.21.15.146:6789/0 conn(0x7fba24074500 legacy=0x7fba2413cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 >> v1:172.21.15.78:6789/0 conn(0x7fba24137d60 legacy=0x7fba24138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 monclient: found mon.noname-b 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba24073840 con 0x7fba2413e370 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.169 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 5 monclient: authenticate success, global_id 124301 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fba140016a0 con 0x7fba2413e370 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: dump: 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.170 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fba140018c0 con 0x7fba2413e370 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 1 -- 172.21.15.78:0/979645153 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fba14001c90 con 0x7fba2413e370 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba294f6640 4 set_mon_vals no callback set 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.161+0000 7fba21ffb640 10 monclient: dump: 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.171 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: shutdown 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- 172.21.15.78:0/979645153 >> v1:172.21.15.78:6790/0 conn(0x7fba2413e370 legacy=0x7fba24140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- 172.21.15.78:0/979645153 shutdown_connections 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- 172.21.15.78:0/979645153 >> 172.21.15.78:0/979645153 conn(0x7fba240aa650 msgr2=0x7fba240aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- 172.21.15.78:0/979645153 shutdown_connections 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- 172.21.15.78:0/979645153 wait complete. 2024-09-15T20:46:38.172 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: build_initial_monmap 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: monmap: 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:38.173 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 librados: starting msgr at 2024-09-15T20:46:38.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 librados: starting objecter 2024-09-15T20:46:38.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 Processor -- start 2024-09-15T20:46:38.177 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- start start 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 librados: setting wanted keys 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 librados: calling monclient init 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: init 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient(hunting): picked mon.noname-c con 0x7fba24074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient(hunting): picked mon.noname-a con 0x7fba24137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient(hunting): picked mon.noname-b con 0x7fba2413e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba24139840 con 0x7fba24137d60 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba24139c40 con 0x7fba2413e370 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fba2413b700 con 0x7fba24074500 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient(hunting): authenticate will time out at 2059.054690s 2024-09-15T20:46:38.178 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba227fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fba24137d60 0x7fba24139130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34620/0 (socket says 172.21.15.78:34620) 2024-09-15T20:46:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba227fc640 1 -- 172.21.15.78:0/3535828470 learned_addr learned my addr 172.21.15.78:0/3535828470 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 client.?.objecter ms_handle_connect 0x7fba24137d60 2024-09-15T20:46:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 client.?.objecter ms_handle_connect 0x7fba24074500 2024-09-15T20:46:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2734612669 0 0) 0x7fba2413b700 con 0x7fba24074500 2024-09-15T20:46:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): my global_id is 124304 2024-09-15T20:46:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba10004ec0 con 0x7fba24074500 2024-09-15T20:46:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 client.?.objecter ms_handle_connect 0x7fba2413e370 2024-09-15T20:46:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.181 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1771233770 0 0) 0x7fba10004ec0 con 0x7fba24074500 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fba2413b700 con 0x7fba24074500 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fba1c001bf0 con 0x7fba24074500 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): dump: 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.182 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1553931393 0 0) 0x7fba24139c40 con 0x7fba2413e370 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient(hunting): my global_id is 134179 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba10004ec0 con 0x7fba2413e370 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2956749911 0 0) 0x7fba2413b700 con 0x7fba24074500 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 >> v1:172.21.15.146:6789/0 conn(0x7fba2413e370 legacy=0x7fba241d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.183 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 >> v1:172.21.15.78:6789/0 conn(0x7fba24137d60 legacy=0x7fba24139130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 monclient: found mon.c 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba2413b9e0 con 0x7fba24074500 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 5 monclient: authenticate success, global_id 124304 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: _renew_subs 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.184 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fba2413d9a0 con 0x7fba24074500 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 client.124304.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: _renew_subs 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fba24141170 con 0x7fba24074500 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba294f6640 1 librados: init done 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fba1c004b70 con 0x7fba24074500 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fba1c004f40 con 0x7fba24074500 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba217fa640 4 set_mon_vals no callback set 2024-09-15T20:46:38.185 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.165+0000 7fba0b7fe640 10 monclient: dump: 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba294f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba294f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.186 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba294f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba04005090 con 0x7fba24074500 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fba1c0050e0 con 0x7fba24074500 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(621..621 src has 379..621) ==== 5532+0+0 (unknown 1478495959 0 0) 0x7fba1c04bb30 con 0x7fba24074500 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba0b7fe640 10 client.124304.objecter ms_dispatch 0x7fba24000ba0 osd_map(621..621 src has 379..621) 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba0b7fe640 3 client.124304.objecter handle_osd_map got epochs [621,621] > 0 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba0b7fe640 3 client.124304.objecter handle_osd_map decoding full epoch 621 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba0b7fe640 20 client.124304.objecter dump_active .. 0 homeless 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba227fc640 10 monclient: get_auth_request con 0x7fba1003e120 auth_method 0 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.169+0000 7fba0b7fe640 10 client.124304.objecter ms_handle_connect 0x7fba1003e120 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.173+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fba1c016e00 con 0x7fba24074500 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.173+0000 7fba0b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.187 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.173+0000 7fba0b7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:38.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:46:38.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:46:38.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fba04005380 con 0x7fba24074500 2024-09-15T20:46:38.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba0b7fe640 1 -- 172.21.15.78:0/3535828470 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2151626742) 0x7fba1c016700 con 0x7fba24074500 2024-09-15T20:46:38.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba0b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:46:38.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba0b7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:38.432 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:46:38.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 10 librados: watch_flush enter 2024-09-15T20:46:38.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 10 librados: watch_flush exit 2024-09-15T20:46:38.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 20 client.124304.objecter shutdown clearing up homeless session... 2024-09-15T20:46:38.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 10 client.124304.objecter successfully canceled tick 2024-09-15T20:46:38.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 >> v1:172.21.15.146:6812/435598642 conn(0x7fba1003e120 legacy=0x7fba100405e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 10 monclient: shutdown 2024-09-15T20:46:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 >> v1:172.21.15.78:6790/0 conn(0x7fba24074500 legacy=0x7fba24138a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 shutdown_connections 2024-09-15T20:46:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 >> 172.21.15.78:0/3535828470 conn(0x7fba240aa650 msgr2=0x7fba240a3ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 shutdown_connections 2024-09-15T20:46:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 1 -- 172.21.15.78:0/3535828470 wait complete. 2024-09-15T20:46:38.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.429+0000 7fba294f6640 1 librados: shutdown 2024-09-15T20:46:38.449 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:46:38.450 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:46:38.450 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:46:38.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient: get_monmap_and_config 2024-09-15T20:46:38.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient: build_initial_monmap 2024-09-15T20:46:38.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient: monmap: 2024-09-15T20:46:38.571 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:38.571 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.571 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:38.573420+0000 2024-09-15T20:46:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:38.573420+0000 2024-09-15T20:46:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:38.572 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:38.572 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 Processor -- start 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- start start 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient: init 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient(hunting): picked mon.noname-c con 0x7ff658135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient(hunting): picked mon.noname-a con 0x7ff6580a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient(hunting): picked mon.noname-b con 0x7ff65813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6580745e0 con 0x7ff6580a24e0 2024-09-15T20:46:38.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65806ecf0 con 0x7ff658135b60 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65806efd0 con 0x7ff65813c220 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient(hunting): authenticate will time out at 2059.458099s 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff656d9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff65813c220 0x7ff65813e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:40932/0 (socket says 172.21.15.78:40932) 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff656d9d640 1 -- 172.21.15.78:0/277748876 learned_addr learned my addr 172.21.15.78:0/277748876 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621088755 0 0) 0x7ff65806efd0 con 0x7ff65813c220 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient(hunting): my global_id is 124307 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff640003530 con 0x7ff65813c220 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400955187 0 0) 0x7ff6580745e0 con 0x7ff6580a24e0 2024-09-15T20:46:38.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient(hunting): my global_id is 144099 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff65806efd0 con 0x7ff6580a24e0 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1301086246 0 0) 0x7ff640003530 con 0x7ff65813c220 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 >> v1:172.21.15.146:6789/0 conn(0x7ff658135b60 legacy=0x7ff658135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 >> v1:172.21.15.78:6789/0 conn(0x7ff6580a24e0 legacy=0x7ff65813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 monclient: found mon.noname-b 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff658136650 con 0x7ff65813c220 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff650001a20 con 0x7ff65813c220 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 5 monclient: authenticate success, global_id 124307 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: dump: 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.583 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff650001c40 con 0x7ff65813c220 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 1 -- 172.21.15.78:0/277748876 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff650002050 con 0x7ff65813c220 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 4 set_mon_vals no callback set 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65559a640 10 monclient: dump: 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.584 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 10 monclient: shutdown 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- 172.21.15.78:0/277748876 >> v1:172.21.15.78:6790/0 conn(0x7ff65813c220 legacy=0x7ff65813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- 172.21.15.78:0/277748876 shutdown_connections 2024-09-15T20:46:38.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- 172.21.15.78:0/277748876 >> 172.21.15.78:0/277748876 conn(0x7ff6580a63c0 msgr2=0x7ff6580a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- 172.21.15.78:0/277748876 shutdown_connections 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.569+0000 7ff65cd7a640 1 -- 172.21.15.78:0/277748876 wait complete. 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: build_initial_monmap 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: monmap: 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.586 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 librados: starting msgr at 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 librados: starting objecter 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 Processor -- start 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 -- start start 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 librados: setting wanted keys 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 librados: calling monclient init 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: init 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient(hunting): picked mon.noname-a con 0x7ff6580a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient(hunting): picked mon.noname-c con 0x7ff658135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient(hunting): picked mon.noname-b con 0x7ff65813c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65809d180 con 0x7ff6580a24e0 2024-09-15T20:46:38.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff658130c20 con 0x7ff65813c220 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff65809e2b0 con 0x7ff658135b60 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient(hunting): authenticate will time out at 2059.461410s 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65659c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff6580a24e0 0x7ff65809d490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34646/0 (socket says 172.21.15.78:34646) 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65659c640 1 -- 172.21.15.78:0/822991134 learned_addr learned my addr 172.21.15.78:0/822991134 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff655d9b640 1 --1- 172.21.15.78:0/822991134 >> v1:172.21.15.78:6790/0 conn(0x7ff658135b60 0x7ff65809dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:40948/0 (socket says 172.21.15.78:40948) 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.?.objecter ms_handle_connect 0x7ff658135b60 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.?.objecter ms_handle_connect 0x7ff6580a24e0 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 851169889 0 0) 0x7ff65809e2b0 con 0x7ff658135b60 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): my global_id is 124310 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff638004ec0 con 0x7ff658135b60 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3304944795 0 0) 0x7ff65806ecf0 con 0x7ff6580a24e0 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): my global_id is 144102 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff65809e2b0 con 0x7ff6580a24e0 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.?.objecter ms_handle_connect 0x7ff65813c220 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4268177337 0 0) 0x7ff638004ec0 con 0x7ff658135b60 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff65806ecf0 con 0x7ff658135b60 2024-09-15T20:46:38.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff6480013c0 con 0x7ff658135b60 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): dump: 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:38.590 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1969801943 0 0) 0x7ff65809e2b0 con 0x7ff6580a24e0 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff638004ec0 con 0x7ff6580a24e0 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff64c001920 con 0x7ff6580a24e0 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): dump: 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.591 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1581010481 0 0) 0x7ff658130c20 con 0x7ff65813c220 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): my global_id is 134185 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff65809e2b0 con 0x7ff65813c220 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3317041314 0 0) 0x7ff65806ecf0 con 0x7ff658135b60 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 >> v1:172.21.15.146:6789/0 conn(0x7ff65813c220 legacy=0x7ff6581d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 >> v1:172.21.15.78:6789/0 conn(0x7ff6580a24e0 legacy=0x7ff65809d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 monclient: found mon.c 2024-09-15T20:46:38.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff658130940 con 0x7ff658135b60 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 5 monclient: authenticate success, global_id 124310 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: _renew_subs 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff65813b1b0 con 0x7ff658135b60 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 client.124310.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: _renew_subs 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff658135610 con 0x7ff658135b60 2024-09-15T20:46:38.593 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 librados: init done 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff648003600 con 0x7ff658135b60 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff648003990 con 0x7ff658135b60 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff654d99640 4 set_mon_vals no callback set 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: dump: 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.594 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff63c0065d0 con 0x7ff63c002870 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff648003b30 con 0x7ff658135b60 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.124310.objecter ms_handle_connect 0x7ff63c002870 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.124310.objecter resend_mon_ops 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(622..622 src has 379..622) ==== 5905+0+0 (unknown 46842108 0 0) 0x7ff64804b8d0 con 0x7ff658135b60 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 client.124310.objecter ms_dispatch 0x7ff658000ba0 osd_map(622..622 src has 379..622) 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 3 client.124310.objecter handle_osd_map got epochs [622,622] > 0 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 3 client.124310.objecter handle_osd_map decoding full epoch 622 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 20 client.124310.objecter dump_active .. 0 homeless 2024-09-15T20:46:38.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3313267496 0 0) 0x7ff63c0065d0 con 0x7ff63c002870 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient(hunting): my global_id is 144105 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff65806ecf0 con 0x7ff63c002870 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1820846432 0 0) 0x7ff65806ecf0 con 0x7ff63c002870 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff63c0065d0 con 0x7ff63c002870 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff64c0037c0 con 0x7ff63c002870 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff646ffd640 10 monclient: dump: 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.596 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.573+0000 7ff656d9d640 10 monclient: get_auth_request con 0x7ff63803e2b0 auth_method 0 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.577+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4221899421 0 0) 0x7ff63c0065d0 con 0x7ff63c002870 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.577+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff63c00a170 con 0x7ff63c002870 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.577+0000 7ff646ffd640 10 client.124310.objecter ms_handle_connect 0x7ff63803e2b0 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.577+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff64c0057d0 con 0x7ff63c002870 2024-09-15T20:46:38.597 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.577+0000 7ff646ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.577+0000 7ff646ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.577+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 >> v1:172.21.15.78:6789/0 conn(0x7ff63c002870 legacy=0x7ff63c004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.598 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.577+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 >> v1:172.21.15.78:6789/0 conn(0x7ff63c002870 legacy=0x7ff63c004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:46:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff63c009e60 con 0x7ff63c009120 2024-09-15T20:46:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 client.124310.objecter ms_handle_connect 0x7ff63c009120 2024-09-15T20:46:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 client.124310.objecter resend_mon_ops 2024-09-15T20:46:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1111264229 0 0) 0x7ff63c009e60 con 0x7ff63c009120 2024-09-15T20:46:38.599 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient(hunting): my global_id is 144108 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff63c0065d0 con 0x7ff63c009120 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 541729478 0 0) 0x7ff63c0065d0 con 0x7ff63c009120 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff63c009e60 con 0x7ff63c009120 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff64c003eb0 con 0x7ff63c009120 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient: dump: 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.600 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1671613793 0 0) 0x7ff63c009e60 con 0x7ff63c009120 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff63c00c400 con 0x7ff63c009120 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3970 (unknown 0 0 1416018034) 0x7ff64c005980 con 0x7ff63c009120 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:38.601 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 >> v1:172.21.15.78:6789/0 conn(0x7ff63c009120 legacy=0x7ff63c0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff646ffd640 1 -- 172.21.15.78:0/822991134 >> v1:172.21.15.78:6789/0 conn(0x7ff63c009120 legacy=0x7ff63c0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 60, 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 475, 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:46:38.602 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.603 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.604 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:46:38.605 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.606 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:46:38.607 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.608 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.609 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.610 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.611 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:38.612 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.613 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.613 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.613 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:46:38.613 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:46:38.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 10 librados: watch_flush enter 2024-09-15T20:46:38.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 10 librados: watch_flush exit 2024-09-15T20:46:38.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 20 client.124310.objecter shutdown clearing up homeless session... 2024-09-15T20:46:38.614 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 10 client.124310.objecter successfully canceled tick 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 >> v1:172.21.15.146:6812/435598642 conn(0x7ff63803e2b0 legacy=0x7ff638040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 10 monclient: shutdown 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 >> v1:172.21.15.78:6790/0 conn(0x7ff658135b60 legacy=0x7ff65809dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff656d9d640 1 -- 172.21.15.78:0/822991134 reap_dead start 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff656d9d640 1 -- 172.21.15.78:0/822991134 reap_dead start 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 shutdown_connections 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 >> 172.21.15.78:0/822991134 conn(0x7ff6580a63c0 msgr2=0x7ff65813f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:38.615 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 shutdown_connections 2024-09-15T20:46:38.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 1 -- 172.21.15.78:0/822991134 wait complete. 2024-09-15T20:46:38.616 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.581+0000 7ff65cd7a640 1 librados: shutdown 2024-09-15T20:46:38.616 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:46:38.616 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:46:38.616 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:46:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient: get_monmap_and_config 2024-09-15T20:46:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient: build_initial_monmap 2024-09-15T20:46:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient: monmap: 2024-09-15T20:46:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:38.731345+0000 2024-09-15T20:46:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:38.731345+0000 2024-09-15T20:46:38.729 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:38.730 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.730 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:38.730 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:38.730 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:38.730 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 1 Processor -- start 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 1 -- start start 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient: init 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient(hunting): picked mon.noname-a con 0x7f51a4135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient(hunting): picked mon.noname-b con 0x7f51a40a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient(hunting): picked mon.noname-c con 0x7f51a413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a40745e0 con 0x7f51a4135b60 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a406ecf0 con 0x7f51a413c220 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a406efd0 con 0x7f51a40a24e0 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.725+0000 7f51ac525640 10 monclient(hunting): authenticate will time out at 2059.615912s 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a9ac0640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f51a40a24e0 0x7f51a413ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:40954/0 (socket says 172.21.15.78:40954) 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a9ac0640 1 -- 172.21.15.78:0/864120248 learned_addr learned my addr 172.21.15.78:0/864120248 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3639832737 0 0) 0x7f51a406efd0 con 0x7f51a40a24e0 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient(hunting): my global_id is 124313 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5190003530 con 0x7f51a40a24e0 2024-09-15T20:46:38.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560692512 0 0) 0x7f51a40745e0 con 0x7f51a4135b60 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient(hunting): my global_id is 144111 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51a406efd0 con 0x7f51a4135b60 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2939498031 0 0) 0x7f51a406efd0 con 0x7f51a4135b60 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 >> v1:172.21.15.78:6790/0 conn(0x7f51a40a24e0 legacy=0x7f51a413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 >> v1:172.21.15.146:6789/0 conn(0x7f51a413c220 legacy=0x7f51a413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 monclient: found mon.noname-a 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:46:38.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a4136650 con 0x7f51a4135b60 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f51a0001600 con 0x7f51a4135b60 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: dump: 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.743 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 5 monclient: authenticate success, global_id 144111 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f51a00017a0 con 0x7f51a4135b60 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 4 set_mon_vals no callback set 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 1 -- 172.21.15.78:0/864120248 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f51a0001bb0 con 0x7f51a4135b60 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a92bf640 10 monclient: dump: 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient: shutdown 2024-09-15T20:46:38.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- 172.21.15.78:0/864120248 >> v1:172.21.15.78:6789/0 conn(0x7f51a4135b60 legacy=0x7f51a4135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- 172.21.15.78:0/864120248 shutdown_connections 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- 172.21.15.78:0/864120248 >> 172.21.15.78:0/864120248 conn(0x7f51a40a63c0 msgr2=0x7f51a40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- 172.21.15.78:0/864120248 shutdown_connections 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- 172.21.15.78:0/864120248 wait complete. 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient: build_initial_monmap 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient: monmap: 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:38.746 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 librados: starting msgr at 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 librados: starting objecter 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 Processor -- start 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- start start 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 librados: setting wanted keys 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 librados: calling monclient init 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient: init 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient(hunting): picked mon.noname-b con 0x7f51a40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient(hunting): picked mon.noname-a con 0x7f51a4135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient(hunting): picked mon.noname-c con 0x7f51a413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a413a7c0 con 0x7f51a4135b60 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a4137980 con 0x7f51a40a24e0 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51a40a5130 con 0x7f51a413c220 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51ac525640 10 monclient(hunting): authenticate will time out at 2059.619193s 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a9ac0640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f51a4135b60 0x7f51a4137270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34666/0 (socket says 172.21.15.78:34666) 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f51a9ac0640 1 -- 172.21.15.78:0/3542675209 learned_addr learned my addr 172.21.15.78:0/3542675209 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 client.?.objecter ms_handle_connect 0x7f51a413c220 2024-09-15T20:46:38.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 client.?.objecter ms_handle_connect 0x7f51a4135b60 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2528961780 0 0) 0x7f51a413a7c0 con 0x7f51a4135b60 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): my global_id is 144114 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5188004ec0 con 0x7f51a4135b60 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3744875482 0 0) 0x7f51a40a5130 con 0x7f51a413c220 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): my global_id is 124316 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51a413a7c0 con 0x7f51a413c220 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 client.?.objecter ms_handle_connect 0x7f51a40a24e0 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1886475692 0 0) 0x7f5188004ec0 con 0x7f51a4135b60 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51a40a5130 con 0x7f51a4135b60 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5194001400 con 0x7f51a4135b60 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): dump: 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.750 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2332633047 0 0) 0x7f51a413a7c0 con 0x7f51a413c220 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5188004ec0 con 0x7f51a413c220 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f519c001bc0 con 0x7f51a413c220 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:38.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.729+0000 7f519affd640 10 monclient(hunting): dump: 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1944417823 0 0) 0x7f51a4137980 con 0x7f51a40a24e0 2024-09-15T20:46:38.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): my global_id is 134191 2024-09-15T20:46:38.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51a413a7c0 con 0x7f51a40a24e0 2024-09-15T20:46:38.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 755629416 0 0) 0x7f51a40a5130 con 0x7f51a4135b60 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6790/0 conn(0x7f51a413c220 legacy=0x7f51a41d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.146:6789/0 conn(0x7f51a40a24e0 legacy=0x7f51a4136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 monclient: found mon.a 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51a413a4e0 con 0x7f51a4135b60 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 5 monclient: authenticate success, global_id 144114 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 10 monclient: _renew_subs 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f51a413b470 con 0x7f51a4135b60 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51a9ac0640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6789/0 conn(0x7f51a4135b60 legacy=0x7f51a4137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51a9ac0640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6789/0 conn(0x7f51a4135b60 legacy=0x7f51a4137270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51a9ac0640 1 --1- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6789/0 conn(0x7f51a4135b60 0x7f51a4137270 :-1 s=OPENED pgs=9160 cs=1 l=1).handle_message read tag failed 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51a9ac0640 1 --1- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6789/0 conn(0x7f51a4135b60 0x7f51a4137270 :-1 s=OPENED pgs=9160 cs=1 l=1).fault on lossy channel, failing 2024-09-15T20:46:38.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: ms_handle_reset current mon v1:172.21.15.78:6789/0 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6789/0 conn(0x7f51a4135b60 legacy=0x7f51a4137270 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): picked mon.b con 0x7f51a40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): picked mon.a con 0x7f518800a350 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 10 client.144114.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): picked mon.c con 0x7f51a413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f51a40a5130 con 0x7f518800a350 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f51a4137980 con 0x7f51a40a24e0 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5188013dc0 con 0x7f51a413c220 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 1 librados: init done 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter ms_handle_connect 0x7f518800a350 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter resend_mon_ops 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter ms_handle_connect 0x7f51a413c220 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter resend_mon_ops 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter ms_handle_connect 0x7f51a40a24e0 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter resend_mon_ops 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3075171085 0 0) 0x7f5188013dc0 con 0x7f51a413c220 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f5188013f90 con 0x7f51a413c220 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1469909251 0 0) 0x7f51a40a5130 con 0x7f518800a350 2024-09-15T20:46:38.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _init_auth already have auth, reseting 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6789/0 -- auth(proto 2 132 bytes epoch 0) -- 0x7f5188013dc0 con 0x7f518800a350 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 233+0+0 (unknown 3370966679 0 0) 0x7f5188013dc0 con 0x7f518800a350 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6790/0 conn(0x7f51a413c220 legacy=0x7f5188010160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.146:6789/0 conn(0x7f51a40a24e0 legacy=0x7f5188005740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 monclient: found mon.a 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+}) -- 0x7f5188015180 con 0x7f518800a350 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f51a413b8b0 con 0x7f518800a350 2024-09-15T20:46:38.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5194004c40 con 0x7f518800a350 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: dump: 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.759 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f51a40008e0 con 0x7f51a4135190 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5194004c40 con 0x7f518800a350 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51a8abe640 4 set_mon_vals no callback set 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter ms_handle_connect 0x7f51a4135190 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter resend_mon_ops 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4158999995 0 0) 0x7f51a40008e0 con 0x7f51a4135190 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient(hunting): my global_id is 124319 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5188013dc0 con 0x7f51a4135190 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f5194010cc0 con 0x7f518800a350 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.0 v1:172.21.15.78:6789/0 6 ==== osd_map(622..622 src has 379..622) ==== 5905+0+0 (unknown 46842108 0 0) 0x7f519404b870 con 0x7f518800a350 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter ms_dispatch 0x7f51a4000ba0 osd_map(622..622 src has 379..622) 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 3 client.144114.objecter handle_osd_map got epochs [622,622] > 0 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 3 client.144114.objecter handle_osd_map decoding full epoch 622 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 20 client.144114.objecter dump_active .. 0 homeless 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 38495392 0 0) 0x7f5188013dc0 con 0x7f51a4135190 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51a40008e0 con 0x7f51a4135190 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f51a0001680 con 0x7f51a4135190 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: dump: 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 4047260150 0 0) 0x7f51a40008e0 con 0x7f51a4135190 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51a409d180 con 0x7f51a4135190 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f51aaac2640 10 monclient: get_auth_request con 0x7f51a4135b60 auth_method 0 2024-09-15T20:46:38.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f51a0001940 con 0x7f51a4135190 2024-09-15T20:46:38.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:38.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6790/0 conn(0x7f51a4135190 legacy=0x7f51a4136b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6790/0 conn(0x7f51a4135190 legacy=0x7f51a4136b60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:38.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.733+0000 7f519affd640 10 client.144114.objecter ms_handle_connect 0x7f51a4135b60 2024-09-15T20:46:38.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f51a413a020 con 0x7f51a4137270 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 client.144114.objecter ms_handle_connect 0x7f51a4137270 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 client.144114.objecter resend_mon_ops 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 83895487 0 0) 0x7f51a413a020 con 0x7f51a4137270 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient(hunting): my global_id is 124322 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51a40008e0 con 0x7f51a4137270 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1429283963 0 0) 0x7f51a40008e0 con 0x7f51a4137270 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51a413a020 con 0x7f51a4137270 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f51a0003820 con 0x7f51a4137270 2024-09-15T20:46:38.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient: dump: 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3232985084 0 0) 0x7f51a413a020 con 0x7f51a4137270 2024-09-15T20:46:38.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51a4070a80 con 0x7f51a4137270 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 301454520) 0x7f51a0005a40 con 0x7f51a4137270 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6790/0 conn(0x7f51a4137270 legacy=0x7f51a402d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f519affd640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6790/0 conn(0x7f51a4137270 legacy=0x7f51a402d450 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:46:38.767 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 60, 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 216669, 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.768 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.769 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.770 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:46:38.771 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.772 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.773 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:46:38.774 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.776 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:38.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:46:38.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:46:38.779 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 10 librados: watch_flush enter 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 10 librados: watch_flush exit 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 20 client.144114.objecter shutdown clearing up homeless session... 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 10 client.144114.objecter successfully canceled tick 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.146:6812/435598642 conn(0x7f51a4135b60 legacy=0x7f518804b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 10 monclient: shutdown 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 >> v1:172.21.15.78:6789/0 conn(0x7f518800a350 legacy=0x7f518800c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51aaac2640 1 -- 172.21.15.78:0/3542675209 reap_dead start 2024-09-15T20:46:38.781 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51aaac2640 1 -- 172.21.15.78:0/3542675209 reap_dead start 2024-09-15T20:46:38.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 shutdown_connections 2024-09-15T20:46:38.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 >> 172.21.15.78:0/3542675209 conn(0x7f51a40a63c0 msgr2=0x7f51a40a4670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:38.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 shutdown_connections 2024-09-15T20:46:38.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 1 -- 172.21.15.78:0/3542675209 wait complete. 2024-09-15T20:46:38.782 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.741+0000 7f51ac525640 1 librados: shutdown 2024-09-15T20:46:38.782 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:46:38.782 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:46:38.782 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:46:38.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.893+0000 7f960929c640 10 monclient: get_monmap_and_config 2024-09-15T20:46:38.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.893+0000 7f960929c640 10 monclient: build_initial_monmap 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.893+0000 7f960929c640 10 monclient: monmap: 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:38.900870+0000 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:38.900870+0000 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:38.899 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 Processor -- start 2024-09-15T20:46:38.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- start start 2024-09-15T20:46:38.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient: init 2024-09-15T20:46:38.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient(hunting): picked mon.noname-a con 0x7f9604074630 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient(hunting): picked mon.noname-c con 0x7f960409d950 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient(hunting): picked mon.noname-b con 0x7f9604074a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9604134bc0 con 0x7f9604074630 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960409f900 con 0x7f960409d950 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960409fbe0 con 0x7f9604074a10 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient(hunting): authenticate will time out at 2059.785371s 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9602ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9604074630 0x7f9604133da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34684/0 (socket says 172.21.15.78:34684) 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f96037fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f9604074a10 0x7f960413d150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41000/0 (socket says 172.21.15.78:41000) 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9602ffd640 1 -- 172.21.15.78:0/1820891215 learned_addr learned my addr 172.21.15.78:0/1820891215 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2306787473 0 0) 0x7f9604134bc0 con 0x7f9604074630 2024-09-15T20:46:38.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient(hunting): my global_id is 144117 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95ec003530 con 0x7f9604074630 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 680864651 0 0) 0x7f960409fbe0 con 0x7f9604074a10 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient(hunting): my global_id is 124325 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9604134bc0 con 0x7f9604074a10 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3190658370 0 0) 0x7f9604134bc0 con 0x7f9604074a10 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 >> v1:172.21.15.146:6789/0 conn(0x7f960409d950 legacy=0x7f96041344b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 >> v1:172.21.15.78:6789/0 conn(0x7f9604074630 legacy=0x7f9604133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 monclient: found mon.noname-b 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960409fec0 con 0x7f9604074a10 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95fc001980 con 0x7f9604074a10 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 5 monclient: authenticate success, global_id 124325 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.910 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: dump: 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95fc001b20 con 0x7f9604074a10 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:38.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 1 -- 172.21.15.78:0/1820891215 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95fc001ef0 con 0x7f9604074a10 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 4 set_mon_vals no callback set 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f9601ffb640 10 monclient: dump: 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.912 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient: shutdown 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- 172.21.15.78:0/1820891215 >> v1:172.21.15.78:6790/0 conn(0x7f9604074a10 legacy=0x7f960413d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- 172.21.15.78:0/1820891215 shutdown_connections 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- 172.21.15.78:0/1820891215 >> 172.21.15.78:0/1820891215 conn(0x7f96040a63c0 msgr2=0x7f96040a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- 172.21.15.78:0/1820891215 shutdown_connections 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 -- 172.21.15.78:0/1820891215 wait complete. 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient: build_initial_monmap 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 10 monclient: monmap: 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:38.913 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 librados: starting msgr at 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.897+0000 7f960929c640 1 librados: starting objecter 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 Processor -- start 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 -- start start 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 librados: setting wanted keys 2024-09-15T20:46:38.914 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 librados: calling monclient init 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient: init 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient(hunting): picked mon.noname-c con 0x7f9604074630 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient(hunting): picked mon.noname-a con 0x7f9604074a10 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient(hunting): picked mon.noname-b con 0x7f960409d950 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96041385f0 con 0x7f9604074a10 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960413a0b0 con 0x7f960409d950 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f960412e4b0 con 0x7f9604074630 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient(hunting): authenticate will time out at 2059.788556s 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f9602ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f9604074630 0x7f960412dda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41014/0 (socket says 172.21.15.78:41014) 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f9602ffd640 1 -- 172.21.15.78:0/4134688900 learned_addr learned my addr 172.21.15.78:0/4134688900 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:38.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.?.objecter ms_handle_connect 0x7f9604074630 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.?.objecter ms_handle_connect 0x7f9604074a10 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4096117703 0 0) 0x7f960412e4b0 con 0x7f9604074630 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): my global_id is 124328 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95f0004ec0 con 0x7f9604074630 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2481389298 0 0) 0x7f96041385f0 con 0x7f9604074a10 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): my global_id is 144120 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f960412e4b0 con 0x7f9604074a10 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.?.objecter ms_handle_connect 0x7f960409d950 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 875277268 0 0) 0x7f960412e4b0 con 0x7f9604074a10 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96041385f0 con 0x7f9604074a10 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2534443584 0 0) 0x7f95f0004ec0 con 0x7f9604074630 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f960412e4b0 con 0x7f9604074630 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95f80018c0 con 0x7f9604074a10 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): dump: 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.917 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95f4001400 con 0x7f9604074630 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): dump: 2024-09-15T20:46:38.918 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4054854840 0 0) 0x7f960413a0b0 con 0x7f960409d950 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): my global_id is 134197 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95f0004ec0 con 0x7f960409d950 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2087160984 0 0) 0x7f96041385f0 con 0x7f9604074a10 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 >> v1:172.21.15.78:6790/0 conn(0x7f9604074630 legacy=0x7f960412dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 >> v1:172.21.15.146:6789/0 conn(0x7f960409d950 legacy=0x7f96041d50f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 monclient: found mon.a 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9604139150 con 0x7f9604074a10 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: _finish_auth 0 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:38.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 5 monclient: authenticate success, global_id 144120 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient: _renew_subs 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f960413b330 con 0x7f9604074a10 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 client.144120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient: _renew_subs 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9604137da0 con 0x7f9604074a10 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 librados: init done 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f95f8003620 con 0x7f9604074a10 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95f8003970 con 0x7f9604074a10 2024-09-15T20:46:38.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: dump: 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f96017fa640 4 set_mon_vals no callback set 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f95e40065d0 con 0x7f95e4002870 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f95f800f9e0 con 0x7f9604074a10 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(622..622 src has 379..622) ==== 5905+0+0 (unknown 46842108 0 0) 0x7f95f804a760 con 0x7f9604074a10 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.144120.objecter ms_dispatch 0x7f9604000ba0 osd_map(622..622 src has 379..622) 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 3 client.144120.objecter handle_osd_map got epochs [622,622] > 0 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 3 client.144120.objecter handle_osd_map decoding full epoch 622 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 20 client.144120.objecter dump_active .. 0 homeless 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.144120.objecter ms_handle_connect 0x7f95e4002870 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.144120.objecter resend_mon_ops 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f96037fe640 10 monclient: get_auth_request con 0x7f95f003e270 auth_method 0 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 412297993 0 0) 0x7f95e40065d0 con 0x7f95e4002870 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient(hunting): my global_id is 134200 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96041385f0 con 0x7f95e4002870 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 client.144120.objecter ms_handle_connect 0x7f95f003e270 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4118636061 0 0) 0x7f96041385f0 con 0x7f95e4002870 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95e40065d0 con 0x7f95e4002870 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95f4004c40 con 0x7f95e4002870 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 10 monclient: dump: 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.924 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2539420390 0 0) 0x7f95e40065d0 con 0x7f95e4002870 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.901+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f95e400a170 con 0x7f95e4002870 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.905+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f95f40061c0 con 0x7f95e4002870 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.905+0000 7f95eb7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.905+0000 7f95eb7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.905+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 >> v1:172.21.15.146:6789/0 conn(0x7f95e4002870 legacy=0x7f95e4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.905+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 >> v1:172.21.15.146:6789/0 conn(0x7f95e4002870 legacy=0x7f95e4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f960929c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f95e4009e60 con 0x7f95e4009120 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f95eb7fe640 10 client.144120.objecter ms_handle_connect 0x7f95e4009120 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f95eb7fe640 10 client.144120.objecter resend_mon_ops 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101195296 0 0) 0x7f95e4009e60 con 0x7f95e4009120 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f95eb7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f95eb7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f95eb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f95eb7fe640 10 monclient(hunting): my global_id is 134203 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.909+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95e40065d0 con 0x7f95e4009120 2024-09-15T20:46:38.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2673886742 0 0) 0x7f95e40065d0 con 0x7f95e4009120 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95e4009e60 con 0x7f95e4009120 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95f4006840 con 0x7f95e4009120 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 10 monclient: dump: 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:38.928 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:38.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2563506676 0 0) 0x7f95e4009e60 con 0x7f95e4009120 2024-09-15T20:46:38.932 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f95e400c3b0 con 0x7f95e4009120 2024-09-15T20:46:38.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 2144881226) 0x7f95f4006660 con 0x7f95e4009120 2024-09-15T20:46:38.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:46:38.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:38.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 >> v1:172.21.15.146:6789/0 conn(0x7f95e4009120 legacy=0x7f95e40095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f95eb7fe640 1 -- 172.21.15.78:0/4134688900 >> v1:172.21.15.146:6789/0 conn(0x7f95e4009120 legacy=0x7f95e40095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:38.933 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:46:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:46:38.933 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 60, 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 108786, 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:46:38.934 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:46:38.935 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.936 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:46:38.937 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:38.938 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.939 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.940 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.941 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.942 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.943 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:38.944 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:38.945 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:46:38.945 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:46:38.946 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 10 librados: watch_flush enter 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 10 librados: watch_flush exit 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 20 client.144120.objecter shutdown clearing up homeless session... 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 10 client.144120.objecter successfully canceled tick 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 >> v1:172.21.15.146:6812/435598642 conn(0x7f95f003e270 legacy=0x7f95f0040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 10 monclient: shutdown 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 >> v1:172.21.15.78:6789/0 conn(0x7f9604074a10 legacy=0x7f96041d19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f96037fe640 1 -- 172.21.15.78:0/4134688900 reap_dead start 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f96037fe640 1 -- 172.21.15.78:0/4134688900 reap_dead start 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 shutdown_connections 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 >> 172.21.15.78:0/4134688900 conn(0x7f96040a63c0 msgr2=0x7f9604130810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 shutdown_connections 2024-09-15T20:46:38.947 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 1 -- 172.21.15.78:0/4134688900 wait complete. 2024-09-15T20:46:38.948 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:38.913+0000 7f960929c640 1 librados: shutdown 2024-09-15T20:46:38.948 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:46:38.948 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:46:38.948 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:46:39.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: get_monmap_and_config 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: build_initial_monmap 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: monmap: 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:39.061085+0000 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:39.061085+0000 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:39.059 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:39.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 Processor -- start 2024-09-15T20:46:39.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- start start 2024-09-15T20:46:39.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: init 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient(hunting): picked mon.noname-b con 0x7f1db4137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient(hunting): picked mon.noname-c con 0x7f1db40a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient(hunting): picked mon.noname-a con 0x7f1db413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db40a6420 con 0x7f1db413e5f0 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db40a6700 con 0x7f1db40a9610 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db40a2110 con 0x7f1db4137ce0 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient(hunting): authenticate will time out at 2059.945729s 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db259c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1db4137ce0 0x7f1db41380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41016/0 (socket says 172.21.15.78:41016) 2024-09-15T20:46:39.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db259c640 1 -- 172.21.15.78:0/1011914999 learned_addr learned my addr 172.21.15.78:0/1011914999 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2788381578 0 0) 0x7f1db40a2110 con 0x7f1db4137ce0 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _negotiate 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): my global_id is 124331 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d9c003530 con 0x7f1db4137ce0 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2963824978 0 0) 0x7f1db40a6420 con 0x7f1db413e5f0 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _negotiate 2024-09-15T20:46:39.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): my global_id is 144123 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1db40a2110 con 0x7f1db413e5f0 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2525567775 0 0) 0x7f1db40a6700 con 0x7f1db40a9610 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _negotiate 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient(hunting): my global_id is 134206 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1db40a6420 con 0x7f1db40a9610 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3940515454 0 0) 0x7f1d9c003530 con 0x7f1db4137ce0 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 >> v1:172.21.15.146:6789/0 conn(0x7f1db40a9610 legacy=0x7f1db413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 >> v1:172.21.15.78:6789/0 conn(0x7f1db413e5f0 legacy=0x7f1db41409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: _finish_hunting 0 2024-09-15T20:46:39.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 monclient: found mon.noname-b 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db40a4bc0 con 0x7f1db4137ce0 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: _finish_auth 0 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: _check_auth_tickets 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1da80016a0 con 0x7f1db4137ce0 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: dump: 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.071 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 5 monclient: authenticate success, global_id 124331 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1da8001840 con 0x7f1db4137ce0 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 1 -- 172.21.15.78:0/1011914999 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1da8001c50 con 0x7f1db4137ce0 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 4 set_mon_vals no callback set 2024-09-15T20:46:39.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db159a640 10 monclient: dump: 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:39.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: shutdown 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- 172.21.15.78:0/1011914999 >> v1:172.21.15.78:6790/0 conn(0x7f1db4137ce0 legacy=0x7f1db41380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- 172.21.15.78:0/1011914999 shutdown_connections 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- 172.21.15.78:0/1011914999 >> 172.21.15.78:0/1011914999 conn(0x7f1db40aa6c0 msgr2=0x7f1db40aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- 172.21.15.78:0/1011914999 shutdown_connections 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 1 -- 172.21.15.78:0/1011914999 wait complete. 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: build_initial_monmap 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.057+0000 7f1db8b16640 10 monclient: monmap: 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:39.074 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 librados: starting msgr at 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 librados: starting objecter 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 Processor -- start 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 -- start start 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 librados: setting wanted keys 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 librados: calling monclient init 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: init 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient(hunting): picked mon.noname-b con 0x7f1db40a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient(hunting): picked mon.noname-c con 0x7f1db4137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:39.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient(hunting): picked mon.noname-a con 0x7f1db413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db413c1a0 con 0x7f1db413e5f0 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db4139cd0 con 0x7f1db40a9610 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1db40a9220 con 0x7f1db4137ce0 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient(hunting): authenticate will time out at 2059.948915s 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db1d9b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1db4137ce0 0x7f1db41395c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:41024/0 (socket says 172.21.15.78:41024) 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db1d9b640 1 -- 172.21.15.78:0/590358755 learned_addr learned my addr 172.21.15.78:0/590358755 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db2d9d640 1 --1- 172.21.15.78:0/590358755 >> v1:172.21.15.78:6789/0 conn(0x7f1db413e5f0 0x7f1db41d4fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34716/0 (socket says 172.21.15.78:34716) 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 client.?.objecter ms_handle_connect 0x7f1db4137ce0 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 client.?.objecter ms_handle_connect 0x7f1db413e5f0 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3440999020 0 0) 0x7f1db40a9220 con 0x7f1db4137ce0 2024-09-15T20:46:39.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): my global_id is 124334 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1d94004ec0 con 0x7f1db4137ce0 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1342539029 0 0) 0x7f1db413c1a0 con 0x7f1db413e5f0 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): my global_id is 144126 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1db40a9220 con 0x7f1db413e5f0 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 client.?.objecter ms_handle_connect 0x7f1db40a9610 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:39.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3038102864 0 0) 0x7f1d94004ec0 con 0x7f1db4137ce0 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1db413c1a0 con 0x7f1db4137ce0 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1da4001980 con 0x7f1db4137ce0 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): dump: 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:39.078 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3298050882 0 0) 0x7f1db40a9220 con 0x7f1db413e5f0 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1d94004ec0 con 0x7f1db413e5f0 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1dac0015a0 con 0x7f1db413e5f0 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): dump: 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:39.079 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2190760945 0 0) 0x7f1db4139cd0 con 0x7f1db40a9610 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient(hunting): my global_id is 134209 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1db40a9220 con 0x7f1db40a9610 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 6746071 0 0) 0x7f1db413c1a0 con 0x7f1db4137ce0 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 >> v1:172.21.15.146:6789/0 conn(0x7f1db40a9610 legacy=0x7f1db4138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 >> v1:172.21.15.78:6789/0 conn(0x7f1db413e5f0 legacy=0x7f1db41d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:39.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 monclient: found mon.c 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1db40a00d0 con 0x7f1db4137ce0 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient: _finish_auth 0 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 5 monclient: authenticate success, global_id 124334 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: _renew_subs 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1db413b330 con 0x7f1db4137ce0 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 client.124334.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:39.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: _renew_subs 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1db413a9a0 con 0x7f1db4137ce0 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 librados: init done 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1da4001ca0 con 0x7f1db4137ce0 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1da4003a70 con 0x7f1db4137ce0 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db0d99640 4 set_mon_vals no callback set 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 monclient: dump: 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:39.082 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d98005090 con 0x7f1db4137ce0 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1da4003c90 con 0x7f1db4137ce0 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(622..622 src has 379..622) ==== 5905+0+0 (unknown 46842108 0 0) 0x7f1da404bb20 con 0x7f1db4137ce0 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 client.124334.objecter ms_dispatch 0x7f1db4000ba0 osd_map(622..622 src has 379..622) 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 3 client.124334.objecter handle_osd_map got epochs [622,622] > 0 2024-09-15T20:46:39.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 3 client.124334.objecter handle_osd_map decoding full epoch 622 2024-09-15T20:46:39.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 20 client.124334.objecter dump_active .. 0 homeless 2024-09-15T20:46:39.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1db259c640 10 monclient: get_auth_request con 0x7f1d9403e2b0 auth_method 0 2024-09-15T20:46:39.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.061+0000 7f1da2ffd640 10 client.124334.objecter ms_handle_connect 0x7f1d9403e2b0 2024-09-15T20:46:39.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.065+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f1da4016c10 con 0x7f1db4137ce0 2024-09-15T20:46:39.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.065+0000 7f1da2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:39.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.065+0000 7f1da2ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:39.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.297+0000 7f1db8b16640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:46:39.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.297+0000 7f1db8b16640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:46:39.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.297+0000 7f1db8b16640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:39.300 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.297+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1d98005c50 con 0x7f1db4137ce0 2024-09-15T20:46:39.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.301+0000 7f1da2ffd640 1 -- 172.21.15.78:0/590358755 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f1da4016510 con 0x7f1db4137ce0 2024-09-15T20:46:39.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.301+0000 7f1da2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:46:39.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.301+0000 7f1da2ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:39.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 10 librados: watch_flush enter 2024-09-15T20:46:39.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 10 librados: watch_flush exit 2024-09-15T20:46:39.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 20 client.124334.objecter shutdown clearing up homeless session... 2024-09-15T20:46:39.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 10 client.124334.objecter successfully canceled tick 2024-09-15T20:46:39.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 >> v1:172.21.15.146:6812/435598642 conn(0x7f1d9403e2b0 legacy=0x7f1d94040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:39.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 10 monclient: shutdown 2024-09-15T20:46:39.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:39.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 >> v1:172.21.15.78:6790/0 conn(0x7f1db4137ce0 legacy=0x7f1db41395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:39.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 shutdown_connections 2024-09-15T20:46:39.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 >> 172.21.15.78:0/590358755 conn(0x7f1db40aa6c0 msgr2=0x7f1db41409e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:39.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.305+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 shutdown_connections 2024-09-15T20:46:39.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.309+0000 7f1db8b16640 1 -- 172.21.15.78:0/590358755 wait complete. 2024-09-15T20:46:39.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:39.309+0000 7f1db8b16640 1 librados: shutdown 2024-09-15T20:46:39.317 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.313+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.313+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=11,logm=53} crc {auth=2529450018,config=2586702516,health=3176145467,kv=731607056,logm=3660518659}) 2024-09-15T20:46:39.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.313+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=2529450018,config=2586702516,health=446239286,kv=199072906,logm=2395968550}) 2024-09-15T20:46:39.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.313+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.313+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=23,config=2,health=11,kv=11,logm=53} crc {auth=2529450018,config=2586702516,health=3176145467,kv=731607056,logm=3660518659}) 2024-09-15T20:46:39.318 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.313+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=22,config=2,health=11,kv=11,logm=54} crc {auth=3392007140,config=2586702516,health=106149319,kv=3001349027,logm=15988127}) 2024-09-15T20:46:39.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.317+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.317+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552840519}) 2024-09-15T20:46:39.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.317+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3324684191}) 2024-09-15T20:46:39.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.317+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.323 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.317+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1552840519}) 2024-09-15T20:46:39.324 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.317+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1231195422}) 2024-09-15T20:46:39.327 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.327 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987058688}) 2024-09-15T20:46:39.327 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=565594048}) 2024-09-15T20:46:39.327 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.327 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2987058688}) 2024-09-15T20:46:39.327 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=520852427}) 2024-09-15T20:46:39.330 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.330 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998730929}) 2024-09-15T20:46:39.330 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4045253184}) 2024-09-15T20:46:39.330 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.330 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=998730929}) 2024-09-15T20:46:39.330 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.325+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=96334946}) 2024-09-15T20:46:39.335 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:46:39.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.329+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.329+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62876598}) 2024-09-15T20:46:39.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.329+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=257427936}) 2024-09-15T20:46:39.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.329+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.329+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=62876598}) 2024-09-15T20:46:39.336 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.329+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1061378111}) 2024-09-15T20:46:39.337 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.337 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960769526}) 2024-09-15T20:46:39.337 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1173116081}) 2024-09-15T20:46:39.337 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.337 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=960769526}) 2024-09-15T20:46:39.337 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=465466128}) 2024-09-15T20:46:39.344 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.344 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167169885}) 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3026302825}) 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2167169885}) 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.333+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2529938016}) 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2247471707}) 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2022718150}) 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2247471707}) 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2471686975}) 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.345 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911688850}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3465196731}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=911688850}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.337+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=112736939}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912279285}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3738066910}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=912279285}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168589235}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975172210}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3767215914}) 2024-09-15T20:46:39.346 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.347 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3975172210}) 2024-09-15T20:46:39.347 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3039302789}) 2024-09-15T20:46:39.348 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.348 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448056999}) 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3816778134}) 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448056999}) 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.341+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=853625505}) 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1061765855}) 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3158566660}) 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1061765855}) 2024-09-15T20:46:39.349 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2581077010}) 2024-09-15T20:46:39.350 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.350 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911811506}) 2024-09-15T20:46:39.350 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2499840906}) 2024-09-15T20:46:39.351 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.351 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911811506}) 2024-09-15T20:46:39.351 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.345+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1316277567}) 2024-09-15T20:46:39.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417722537}) 2024-09-15T20:46:39.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=274364254}) 2024-09-15T20:46:39.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=417722537}) 2024-09-15T20:46:39.352 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2485231767}) 2024-09-15T20:46:39.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005713740}) 2024-09-15T20:46:39.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2150076434}) 2024-09-15T20:46:39.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4005713740}) 2024-09-15T20:46:39.354 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.349+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=138075263}) 2024-09-15T20:46:39.356 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.356 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403239374}) 2024-09-15T20:46:39.356 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=102919898}) 2024-09-15T20:46:39.356 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.356 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=403239374}) 2024-09-15T20:46:39.356 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2903459521}) 2024-09-15T20:46:39.358 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.358 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143923074}) 2024-09-15T20:46:39.358 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1309118336}) 2024-09-15T20:46:39.358 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.358 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2143923074}) 2024-09-15T20:46:39.358 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.353+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1705982147}) 2024-09-15T20:46:39.360 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.360 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311860774}) 2024-09-15T20:46:39.360 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3782597196}) 2024-09-15T20:46:39.360 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.360 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3311860774}) 2024-09-15T20:46:39.360 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3029997536}) 2024-09-15T20:46:39.362 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.362 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2567977541}) 2024-09-15T20:46:39.362 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2440402808}) 2024-09-15T20:46:39.362 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2567977541}) 2024-09-15T20:46:39.363 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.357+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=987229675}) 2024-09-15T20:46:39.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504467965}) 2024-09-15T20:46:39.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1378362377}) 2024-09-15T20:46:39.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1504467965}) 2024-09-15T20:46:39.364 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2540677765}) 2024-09-15T20:46:39.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448983233}) 2024-09-15T20:46:39.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2565032665}) 2024-09-15T20:46:39.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2448983233}) 2024-09-15T20:46:39.366 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.361+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1800291518}) 2024-09-15T20:46:39.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594909184}) 2024-09-15T20:46:39.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=453968990}) 2024-09-15T20:46:39.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1594909184}) 2024-09-15T20:46:39.368 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=426696493}) 2024-09-15T20:46:39.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601739490}) 2024-09-15T20:46:39.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2620910244}) 2024-09-15T20:46:39.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3601739490}) 2024-09-15T20:46:39.370 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.365+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612853447}) 2024-09-15T20:46:39.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2475921729}) 2024-09-15T20:46:39.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1403756740}) 2024-09-15T20:46:39.371 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.372 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2475921729}) 2024-09-15T20:46:39.372 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1901202845}) 2024-09-15T20:46:39.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002336312}) 2024-09-15T20:46:39.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1689693258}) 2024-09-15T20:46:39.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.373 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4002336312}) 2024-09-15T20:46:39.374 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.369+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4284534405}) 2024-09-15T20:46:39.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955275215}) 2024-09-15T20:46:39.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3195667997}) 2024-09-15T20:46:39.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=955275215}) 2024-09-15T20:46:39.375 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3713274451}) 2024-09-15T20:46:39.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214007882}) 2024-09-15T20:46:39.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=946767411}) 2024-09-15T20:46:39.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=214007882}) 2024-09-15T20:46:39.377 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3571279751}) 2024-09-15T20:46:39.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037876463}) 2024-09-15T20:46:39.378 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.373+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3091327445}) 2024-09-15T20:46:39.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1037876463}) 2024-09-15T20:46:39.379 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2249656673}) 2024-09-15T20:46:39.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.380 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004537246}) 2024-09-15T20:46:39.381 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2369911479}) 2024-09-15T20:46:39.381 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.381 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2004537246}) 2024-09-15T20:46:39.381 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3893401437}) 2024-09-15T20:46:39.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723128218}) 2024-09-15T20:46:39.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1655972934}) 2024-09-15T20:46:39.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3723128218}) 2024-09-15T20:46:39.382 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.377+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=292219084}) 2024-09-15T20:46:39.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385018186}) 2024-09-15T20:46:39.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2961065154}) 2024-09-15T20:46:39.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3385018186}) 2024-09-15T20:46:39.384 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=885627256}) 2024-09-15T20:46:39.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825525981}) 2024-09-15T20:46:39.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=147553622}) 2024-09-15T20:46:39.385 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1825525981}) 2024-09-15T20:46:39.386 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.381+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2322264475}) 2024-09-15T20:46:39.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632395888}) 2024-09-15T20:46:39.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2635081083}) 2024-09-15T20:46:39.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3632395888}) 2024-09-15T20:46:39.387 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3757241500}) 2024-09-15T20:46:39.388 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.388 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1116731919}) 2024-09-15T20:46:39.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3457699304}) 2024-09-15T20:46:39.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1116731919}) 2024-09-15T20:46:39.389 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4003680859}) 2024-09-15T20:46:39.390 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1820005646}) 2024-09-15T20:46:39.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1112785516}) 2024-09-15T20:46:39.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1820005646}) 2024-09-15T20:46:39.391 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.385+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4129760667}) 2024-09-15T20:46:39.392 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.392 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693730109}) 2024-09-15T20:46:39.392 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1742222431}) 2024-09-15T20:46:39.392 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.392 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=693730109}) 2024-09-15T20:46:39.392 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3943598705}) 2024-09-15T20:46:39.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.393 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3721978988}) 2024-09-15T20:46:39.394 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4125980587}) 2024-09-15T20:46:39.394 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.394 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3721978988}) 2024-09-15T20:46:39.394 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.389+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3522893796}) 2024-09-15T20:46:39.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3735376990}) 2024-09-15T20:46:39.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=852477598}) 2024-09-15T20:46:39.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3735376990}) 2024-09-15T20:46:39.395 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=181474269}) 2024-09-15T20:46:39.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488631931}) 2024-09-15T20:46:39.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3451751415}) 2024-09-15T20:46:39.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2488631931}) 2024-09-15T20:46:39.397 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=869344914}) 2024-09-15T20:46:39.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349330007}) 2024-09-15T20:46:39.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2154461125}) 2024-09-15T20:46:39.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3349330007}) 2024-09-15T20:46:39.398 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.393+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2756040014}) 2024-09-15T20:46:39.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920911864}) 2024-09-15T20:46:39.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4091614313}) 2024-09-15T20:46:39.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2920911864}) 2024-09-15T20:46:39.400 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1324727413}) 2024-09-15T20:46:39.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032867495}) 2024-09-15T20:46:39.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3694560332}) 2024-09-15T20:46:39.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1032867495}) 2024-09-15T20:46:39.402 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.397+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2816485224}) 2024-09-15T20:46:39.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699567179}) 2024-09-15T20:46:39.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4089629433}) 2024-09-15T20:46:39.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=699567179}) 2024-09-15T20:46:39.403 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3256248628}) 2024-09-15T20:46:39.405 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.405 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3055425746}) 2024-09-15T20:46:39.405 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1028929012}) 2024-09-15T20:46:39.405 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.405 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3055425746}) 2024-09-15T20:46:39.405 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.401+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3084029227}) 2024-09-15T20:46:39.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426286626}) 2024-09-15T20:46:39.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2780684891}) 2024-09-15T20:46:39.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1426286626}) 2024-09-15T20:46:39.407 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3635496112}) 2024-09-15T20:46:39.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513849752}) 2024-09-15T20:46:39.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3513567545}) 2024-09-15T20:46:39.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1513849752}) 2024-09-15T20:46:39.409 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.405+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=984436624}) 2024-09-15T20:46:39.411 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.411 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764344537}) 2024-09-15T20:46:39.411 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2049112355}) 2024-09-15T20:46:39.411 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.411 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=764344537}) 2024-09-15T20:46:39.411 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=88372753}) 2024-09-15T20:46:39.413 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.413 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264227333}) 2024-09-15T20:46:39.413 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3466912404}) 2024-09-15T20:46:39.413 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.413 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4264227333}) 2024-09-15T20:46:39.413 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.409+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3642830685}) 2024-09-15T20:46:39.415 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.415 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434818872}) 2024-09-15T20:46:39.415 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3204240257}) 2024-09-15T20:46:39.415 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.415 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=434818872}) 2024-09-15T20:46:39.415 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1738966782}) 2024-09-15T20:46:39.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596471742}) 2024-09-15T20:46:39.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1074528165}) 2024-09-15T20:46:39.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2596471742}) 2024-09-15T20:46:39.417 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.413+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1090893672}) 2024-09-15T20:46:39.419 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.419 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578715698}) 2024-09-15T20:46:39.419 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=925784094}) 2024-09-15T20:46:39.420 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.420 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1578715698}) 2024-09-15T20:46:39.420 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2757015912}) 2024-09-15T20:46:39.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1262267838}) 2024-09-15T20:46:39.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2696674286}) 2024-09-15T20:46:39.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1262267838}) 2024-09-15T20:46:39.421 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.417+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=474093068}) 2024-09-15T20:46:39.423 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.423 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921722154}) 2024-09-15T20:46:39.423 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1405035516}) 2024-09-15T20:46:39.423 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.423 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1921722154}) 2024-09-15T20:46:39.423 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1437875272}) 2024-09-15T20:46:39.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076228715}) 2024-09-15T20:46:39.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4064294305}) 2024-09-15T20:46:39.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2076228715}) 2024-09-15T20:46:39.425 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.421+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=136704492}) 2024-09-15T20:46:39.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3557798628}) 2024-09-15T20:46:39.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3887064835}) 2024-09-15T20:46:39.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3557798628}) 2024-09-15T20:46:39.427 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4229442067}) 2024-09-15T20:46:39.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1671642438}) 2024-09-15T20:46:39.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2116114777}) 2024-09-15T20:46:39.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1671642438}) 2024-09-15T20:46:39.429 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.425+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1338205008}) 2024-09-15T20:46:39.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331493741}) 2024-09-15T20:46:39.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2357207140}) 2024-09-15T20:46:39.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3331493741}) 2024-09-15T20:46:39.431 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=759963411}) 2024-09-15T20:46:39.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203767564}) 2024-09-15T20:46:39.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3050062355}) 2024-09-15T20:46:39.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4203767564}) 2024-09-15T20:46:39.433 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.429+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1426846957}) 2024-09-15T20:46:39.435 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.435 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537635559}) 2024-09-15T20:46:39.435 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=414562593}) 2024-09-15T20:46:39.435 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.435 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=537635559}) 2024-09-15T20:46:39.435 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2914765021}) 2024-09-15T20:46:39.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697379442}) 2024-09-15T20:46:39.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3678668268}) 2024-09-15T20:46:39.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1697379442}) 2024-09-15T20:46:39.437 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.433+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2138557712}) 2024-09-15T20:46:39.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759974124}) 2024-09-15T20:46:39.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3591430119,mdsmap=1235320281}) 2024-09-15T20:46:39.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759974124}) 2024-09-15T20:46:39.439 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=99,mdsmap=1} crc {logm=3968833180,mdsmap=1228700967}) 2024-09-15T20:46:39.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=6} crc {logm=1550175786,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=2468677156,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1532307299,osd_pg_creating=4147521026,osd_snap=2479711666,osdmap=2330574322}) 2024-09-15T20:46:39.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=34,osdmap=12} crc {logm=1550175786,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=2161993431,mon_config_key=2636242322,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=4147521026,osd_snap=1296989086,osdmap=1184705247}) 2024-09-15T20:46:39.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=1,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=17,monmap=3,nvmeofgw=2,osd_metadata=6,osd_pg_creating=1,osd_snap=40,osdmap=6} crc {logm=1550175786,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=2468677156,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=1596038039,osd_metadata=1532307299,osd_pg_creating=4147521026,osd_snap=2479711666,osdmap=2330574322}) 2024-09-15T20:46:39.442 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.437+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=1,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=39,osdmap=9} crc {logm=1550175786,mdsmap=503983409,mgr=4145042074,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=346684683,mon_config_key=342708592,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=4147521026,osd_snap=3274185474,osdmap=3997569681}) 2024-09-15T20:46:39.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2859282044}) 2024-09-15T20:46:39.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1709080578}) 2024-09-15T20:46:39.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2859282044}) 2024-09-15T20:46:39.444 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4258721713}) 2024-09-15T20:46:39.446 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.446 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3269847725}) 2024-09-15T20:46:39.446 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3739836046}) 2024-09-15T20:46:39.446 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3269847725}) 2024-09-15T20:46:39.447 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.441+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1456126661}) 2024-09-15T20:46:39.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.445+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.445+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1422959910}) 2024-09-15T20:46:39.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.445+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=925210458}) 2024-09-15T20:46:39.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.445+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.445+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1422959910}) 2024-09-15T20:46:39.450 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.445+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=225316235}) 2024-09-15T20:46:39.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.449+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.452 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.449+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=2885644009}) 2024-09-15T20:46:39.453 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.449+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=58} crc {osdmap=3823799278}) 2024-09-15T20:46:39.453 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.449+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:46:39.453 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.449+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=59} crc {osdmap=2885644009}) 2024-09-15T20:46:39.453 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:46:39.449+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=59} crc {osdmap=1178283740}) 2024-09-15T20:46:40.336 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:46:40.336 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:46:40.463 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient: get_monmap_and_config 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient: build_initial_monmap 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient: monmap: 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:40.466068+0000 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:40.466068+0000 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:40.464 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:40.465 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 1 Processor -- start 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 1 -- start start 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient: init 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient(hunting): picked mon.noname-a con 0x7fb8580a0ff0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient(hunting): picked mon.noname-c con 0x7fb858073800 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient(hunting): picked mon.noname-b con 0x7fb8580a1ae0 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb858072df0 con 0x7fb8580a0ff0 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb858074110 con 0x7fb858073800 2024-09-15T20:46:40.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb8580743f0 con 0x7fb8580a1ae0 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient(hunting): authenticate will time out at 2061.350699s 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85d409640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb8580a1ae0 0x7fb858140400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53146/0 (socket says 172.21.15.78:53146) 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85d409640 1 -- 172.21.15.78:0/3857421068 learned_addr learned my addr 172.21.15.78:0/3857421068 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662087482 0 0) 0x7fb8580743f0 con 0x7fb8580a1ae0 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient(hunting): my global_id is 124337 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb844003530 con 0x7fb8580a1ae0 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 16612739 0 0) 0x7fb858072df0 con 0x7fb8580a0ff0 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient(hunting): my global_id is 144129 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb8580743f0 con 0x7fb8580a0ff0 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3609975956 0 0) 0x7fb844003530 con 0x7fb8580a1ae0 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 >> v1:172.21.15.146:6789/0 conn(0x7fb858073800 legacy=0x7fb85813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 >> v1:172.21.15.78:6789/0 conn(0x7fb8580a0ff0 legacy=0x7fb8580a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: _finish_hunting 0 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 monclient: found mon.noname-b 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb858074eb0 con 0x7fb8580a1ae0 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: _finish_auth 0 2024-09-15T20:46:40.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: _check_auth_tickets 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb8500016a0 con 0x7fb8580a1ae0 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: dump: 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.474 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 5 monclient: authenticate success, global_id 124337 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb8500018c0 con 0x7fb8580a1ae0 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 1 -- 172.21.15.78:0/3857421068 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb850001c90 con 0x7fb8580a1ae0 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb8577fe640 10 monclient: dump: 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.475 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.461+0000 7fb85ee6c640 4 set_mon_vals no callback set 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: shutdown 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3857421068 >> v1:172.21.15.78:6790/0 conn(0x7fb8580a1ae0 legacy=0x7fb858140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3857421068 shutdown_connections 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3857421068 >> 172.21.15.78:0/3857421068 conn(0x7fb8580a63c0 msgr2=0x7fb8580a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3857421068 shutdown_connections 2024-09-15T20:46:40.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3857421068 wait complete. 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: build_initial_monmap 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: monmap: 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 librados: starting msgr at 2024-09-15T20:46:40.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 librados: starting objecter 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 Processor -- start 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- start start 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 librados: setting wanted keys 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 librados: calling monclient init 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: init 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient(hunting): picked mon.noname-c con 0x7fb858073800 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient(hunting): picked mon.noname-b con 0x7fb8580a0ff0 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient(hunting): picked mon.noname-a con 0x7fb8580a1ae0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85811f950 con 0x7fb8580a1ae0 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85811d400 con 0x7fb8580a0ff0 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb85811fc30 con 0x7fb858073800 2024-09-15T20:46:40.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient(hunting): authenticate will time out at 2061.353786s 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85cc08640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb858073800 0x7fb85811c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53150/0 (socket says 172.21.15.78:53150) 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85cc08640 1 -- 172.21.15.78:0/3308526065 learned_addr learned my addr 172.21.15.78:0/3308526065 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 client.?.objecter ms_handle_connect 0x7fb858073800 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 client.?.objecter ms_handle_connect 0x7fb8580a1ae0 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 830748039 0 0) 0x7fb85811fc30 con 0x7fb858073800 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): my global_id is 124340 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb838004ec0 con 0x7fb858073800 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2479229533 0 0) 0x7fb85811f950 con 0x7fb8580a1ae0 2024-09-15T20:46:40.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): my global_id is 144132 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb85811fc30 con 0x7fb8580a1ae0 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 client.?.objecter ms_handle_connect 0x7fb8580a0ff0 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4172814612 0 0) 0x7fb85811fc30 con 0x7fb8580a1ae0 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb85811f950 con 0x7fb8580a1ae0 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb850001a70 con 0x7fb8580a1ae0 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): dump: 2024-09-15T20:46:40.480 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 918436308 0 0) 0x7fb838004ec0 con 0x7fb858073800 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb85811fc30 con 0x7fb858073800 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb848001d20 con 0x7fb858073800 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:40.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): dump: 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2357928088 0 0) 0x7fb85811d400 con 0x7fb8580a0ff0 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient(hunting): my global_id is 134215 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb838004ec0 con 0x7fb8580a0ff0 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 878768743 0 0) 0x7fb85811f950 con 0x7fb8580a1ae0 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 >> v1:172.21.15.78:6790/0 conn(0x7fb858073800 legacy=0x7fb85811c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 >> v1:172.21.15.146:6789/0 conn(0x7fb8580a0ff0 legacy=0x7fb85811ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient: _finish_hunting 0 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 monclient: found mon.a 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb858140e70 con 0x7fb8580a1ae0 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient: _finish_auth 0 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient: _check_auth_tickets 2024-09-15T20:46:40.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 5 monclient: authenticate success, global_id 144132 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: _renew_subs 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb85809dd00 con 0x7fb8580a1ae0 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 client.144132.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: _renew_subs 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb85811e700 con 0x7fb8580a1ae0 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 librados: init done 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb850001d10 con 0x7fb8580a1ae0 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb850004c00 con 0x7fb8580a1ae0 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:40.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 monclient: dump: 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8567fc640 4 set_mon_vals no callback set 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:40.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb83c005090 con 0x7fb8580a1ae0 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb850010c70 con 0x7fb8580a1ae0 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(624..624 src has 379..624) ==== 5913+0+0 (unknown 149201672 0 0) 0x7fb85004b8b0 con 0x7fb8580a1ae0 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 10 client.144132.objecter ms_dispatch 0x7fb858000ba0 osd_map(624..624 src has 379..624) 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 3 client.144132.objecter handle_osd_map got epochs [624,624] > 0 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.465+0000 7fb8557fa640 3 client.144132.objecter handle_osd_map decoding full epoch 624 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.469+0000 7fb8557fa640 20 client.144132.objecter dump_active .. 0 homeless 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.469+0000 7fb85cc08640 10 monclient: get_auth_request con 0x7fb83803e290 auth_method 0 2024-09-15T20:46:40.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.469+0000 7fb8557fa640 10 client.144132.objecter ms_handle_connect 0x7fb83803e290 2024-09-15T20:46:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.473+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb8500169e0 con 0x7fb8580a1ae0 2024-09-15T20:46:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.473+0000 7fb8557fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:40.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.473+0000 7fb8557fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:40.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.705+0000 7fb85ee6c640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:46:40.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.705+0000 7fb85ee6c640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:46:40.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.705+0000 7fb85ee6c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:40.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.705+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb83c005380 con 0x7fb8580a1ae0 2024-09-15T20:46:40.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.705+0000 7fb8557fa640 1 -- 172.21.15.78:0/3308526065 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2140611843) 0x7fb8500162e0 con 0x7fb8580a1ae0 2024-09-15T20:46:40.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.705+0000 7fb8557fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:46:40.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.705+0000 7fb8557fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:40.710 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":60,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:46:40.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 10 librados: watch_flush enter 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 10 librados: watch_flush exit 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 20 client.144132.objecter shutdown clearing up homeless session... 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 10 client.144132.objecter successfully canceled tick 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 >> v1:172.21.15.146:6812/435598642 conn(0x7fb83803e290 legacy=0x7fb838040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 10 monclient: shutdown 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 >> v1:172.21.15.78:6789/0 conn(0x7fb8580a1ae0 legacy=0x7fb8581d4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 shutdown_connections 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 >> 172.21.15.78:0/3308526065 conn(0x7fb8580a63c0 msgr2=0x7fb8580a4cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 shutdown_connections 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 1 -- 172.21.15.78:0/3308526065 wait complete. 2024-09-15T20:46:40.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.709+0000 7fb85ee6c640 1 librados: shutdown 2024-09-15T20:46:40.728 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:46:40.729 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:46:40.729 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:46:40.729 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:46:40.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient: get_monmap_and_config 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient: build_initial_monmap 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient: monmap: 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:40.854085+0000 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:40.854085+0000 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:40.852 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:40.853 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 1 Processor -- start 2024-09-15T20:46:40.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 1 -- start start 2024-09-15T20:46:40.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient: init 2024-09-15T20:46:40.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:40.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:46:40.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient(hunting): picked mon.noname-b con 0x7faac8135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient(hunting): picked mon.noname-a con 0x7faac80a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient(hunting): picked mon.noname-c con 0x7faac813c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac80745e0 con 0x7faac80a24e0 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac806ecf0 con 0x7faac813c220 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac806efd0 con 0x7faac8135b60 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient(hunting): authenticate will time out at 2061.738665s 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7faac80a24e0 0x7faac813ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:53874/0 (socket says 172.21.15.78:53874) 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac77fe640 1 -- 172.21.15.78:0/4288760984 learned_addr learned my addr 172.21.15.78:0/4288760984 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3026889803 0 0) 0x7faac806efd0 con 0x7faac8135b60 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.862 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): my global_id is 124343 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faab4003530 con 0x7faac8135b60 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1325471254 0 0) 0x7faac80745e0 con 0x7faac80a24e0 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): my global_id is 144135 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac806efd0 con 0x7faac80a24e0 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 193759212 0 0) 0x7faac806ecf0 con 0x7faac813c220 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient(hunting): my global_id is 134218 2024-09-15T20:46:40.863 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac80745e0 con 0x7faac813c220 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4155987147 0 0) 0x7faab4003530 con 0x7faac8135b60 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 >> v1:172.21.15.146:6789/0 conn(0x7faac813c220 legacy=0x7faac813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 >> v1:172.21.15.78:6789/0 conn(0x7faac80a24e0 legacy=0x7faac813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 monclient: found mon.noname-b 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac8136650 con 0x7faac8135b60 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: _finish_auth 0 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faab8001a20 con 0x7faac8135b60 2024-09-15T20:46:40.864 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: dump: 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 5 monclient: authenticate success, global_id 124343 2024-09-15T20:46:40.865 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faab8001bc0 con 0x7faac8135b60 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 1 -- 172.21.15.78:0/4288760984 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faab8001f90 con 0x7faac8135b60 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faac6ffd640 10 monclient: dump: 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.866 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.849+0000 7faace2ba640 4 set_mon_vals no callback set 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: shutdown 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- 172.21.15.78:0/4288760984 >> v1:172.21.15.78:6790/0 conn(0x7faac8135b60 legacy=0x7faac8135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- 172.21.15.78:0/4288760984 shutdown_connections 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- 172.21.15.78:0/4288760984 >> 172.21.15.78:0/4288760984 conn(0x7faac80a63c0 msgr2=0x7faac80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- 172.21.15.78:0/4288760984 shutdown_connections 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- 172.21.15.78:0/4288760984 wait complete. 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: build_initial_monmap 2024-09-15T20:46:40.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: monmap: 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 librados: starting msgr at 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 librados: starting objecter 2024-09-15T20:46:40.868 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 Processor -- start 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- start start 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 librados: setting wanted keys 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 librados: calling monclient init 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: init 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient(hunting): picked mon.noname-b con 0x7faac80a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient(hunting): picked mon.noname-a con 0x7faac8135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient(hunting): picked mon.noname-c con 0x7faac813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac8134ed0 con 0x7faac8135b60 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac813bef0 con 0x7faac80a24e0 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faac809d180 con 0x7faac813c220 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:40.869 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient(hunting): authenticate will time out at 2061.742218s 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7faac8135b60 0x7faac81cd6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:53876/0 (socket says 172.21.15.78:53876) 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac77fe640 1 -- 172.21.15.78:0/2953365381 learned_addr learned my addr 172.21.15.78:0/2953365381 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 client.?.objecter ms_handle_connect 0x7faac813c220 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 client.?.objecter ms_handle_connect 0x7faac8135b60 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1088407623 0 0) 0x7faac809d180 con 0x7faac813c220 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): my global_id is 124346 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaa8004ec0 con 0x7faac813c220 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2318064533 0 0) 0x7faac8134ed0 con 0x7faac8135b60 2024-09-15T20:46:40.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): my global_id is 144138 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac809d180 con 0x7faac8135b60 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 client.?.objecter ms_handle_connect 0x7faac80a24e0 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 785704391 0 0) 0x7faac809d180 con 0x7faac8135b60 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faac8134ed0 con 0x7faac8135b60 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faab0001a00 con 0x7faac8135b60 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): dump: 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.871 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4063856907 0 0) 0x7faaa8004ec0 con 0x7faac813c220 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faac809d180 con 0x7faac813c220 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faac0001880 con 0x7faac813c220 2024-09-15T20:46:40.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): dump: 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 764617033 0 0) 0x7faac813bef0 con 0x7faac80a24e0 2024-09-15T20:46:40.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient(hunting): my global_id is 134221 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaa8004ec0 con 0x7faac80a24e0 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4218565415 0 0) 0x7faac8134ed0 con 0x7faac8135b60 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 >> v1:172.21.15.78:6790/0 conn(0x7faac813c220 legacy=0x7faac81d0df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 >> v1:172.21.15.146:6789/0 conn(0x7faac80a24e0 legacy=0x7faac81347c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 monclient: found mon.a 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faac811d2d0 con 0x7faac8135b60 2024-09-15T20:46:40.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient: _finish_auth 0 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 5 monclient: authenticate success, global_id 144138 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: _renew_subs 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7faac811e140 con 0x7faac8135b60 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 client.144138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: _renew_subs 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:40.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7faac81c9220 con 0x7faac8135b60 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 librados: init done 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7faab00047f0 con 0x7faac8135b60 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faab0004b40 con 0x7faac8135b60 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac67fc640 4 set_mon_vals no callback set 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faac4ff9640 10 monclient: dump: 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.876 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.853+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faaac0065d0 con 0x7faaac002870 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 client.144138.objecter ms_handle_connect 0x7faaac002870 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 client.144138.objecter resend_mon_ops 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165992300 0 0) 0x7faaac0065d0 con 0x7faaac002870 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient(hunting): my global_id is 144141 2024-09-15T20:46:40.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faac8134ed0 con 0x7faaac002870 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7faab0010bb0 con 0x7faac8135b60 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(624..624 src has 379..624) ==== 5913+0+0 (unknown 149201672 0 0) 0x7faab004b8e0 con 0x7faac8135b60 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 client.144138.objecter ms_dispatch 0x7faac8000ba0 osd_map(624..624 src has 379..624) 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 3 client.144138.objecter handle_osd_map got epochs [624,624] > 0 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 3 client.144138.objecter handle_osd_map decoding full epoch 624 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 20 client.144138.objecter dump_active .. 0 homeless 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3779118267 0 0) 0x7faac8134ed0 con 0x7faaac002870 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faaac0065d0 con 0x7faaac002870 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faab80018d0 con 0x7faaac002870 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient: dump: 2024-09-15T20:46:40.878 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1817901327 0 0) 0x7faaac0065d0 con 0x7faaac002870 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7faaac00a170 con 0x7faaac002870 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faacc857640 10 monclient: get_auth_request con 0x7faaa803e220 auth_method 0 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7faab8001a10 con 0x7faaac002870 2024-09-15T20:46:40.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:40.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:40.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 >> v1:172.21.15.78:6789/0 conn(0x7faaac002870 legacy=0x7faaac004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 >> v1:172.21.15.78:6789/0 conn(0x7faaac002870 legacy=0x7faaac004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:40.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.857+0000 7faac4ff9640 10 client.144138.objecter ms_handle_connect 0x7faaa803e220 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faace2ba640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7faaac009e80 con 0x7faaac009140 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 client.144138.objecter ms_handle_connect 0x7faaac009140 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 client.144138.objecter resend_mon_ops 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1436704300 0 0) 0x7faaac009e80 con 0x7faaac009140 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 monclient(hunting): my global_id is 144144 2024-09-15T20:46:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7faaac0065d0 con 0x7faaac009140 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3324968948 0 0) 0x7faaac0065d0 con 0x7faaac009140 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7faaac009e80 con 0x7faaac009140 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7faab8003b20 con 0x7faaac009140 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.861+0000 7faac4ff9640 10 monclient: dump: 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.883 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 206532457 0 0) 0x7faaac009e80 con 0x7faaac009140 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7faaac00c400 con 0x7faaac009140 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3971 (unknown 0 0 516351939) 0x7faab8003ab0 con 0x7faaac009140 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faac4ff9640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faac4ff9640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:40.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 >> v1:172.21.15.78:6789/0 conn(0x7faaac009140 legacy=0x7faaac0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faac4ff9640 1 -- 172.21.15.78:0/2953365381 >> v1:172.21.15.78:6789/0 conn(0x7faaac009140 legacy=0x7faaac0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:40.885 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:46:40.885 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:46:40.885 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:46:40.885 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:46:40.885 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 60, 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 2759, 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:40.886 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:46:40.887 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:40.888 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:46:40.889 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:46:40.890 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:46:40.890 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:46:40.890 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:46:40.890 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:40.890 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:46:40.890 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:40.891 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:40.892 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:46:40.893 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:40.894 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:40.894 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:40.894 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:40.894 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:46:40.894 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:40.894 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:40.894 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:40.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.895 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:46:40.895 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:46:40.895 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:40.895 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:40.895 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:40.895 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.895 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:46:40.896 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:40.897 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:46:40.898 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:46:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 10 librados: watch_flush enter 2024-09-15T20:46:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 10 librados: watch_flush exit 2024-09-15T20:46:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 20 client.144138.objecter shutdown clearing up homeless session... 2024-09-15T20:46:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 10 client.144138.objecter successfully canceled tick 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 >> v1:172.21.15.146:6812/435598642 conn(0x7faaa803e220 legacy=0x7faaa80406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 10 monclient: shutdown 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 >> v1:172.21.15.78:6789/0 conn(0x7faac8135b60 legacy=0x7faac81cd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faacc857640 1 -- 172.21.15.78:0/2953365381 reap_dead start 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faacc857640 1 -- 172.21.15.78:0/2953365381 reap_dead start 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 shutdown_connections 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 >> 172.21.15.78:0/2953365381 conn(0x7faac80a63c0 msgr2=0x7faac80a45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 shutdown_connections 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 1 -- 172.21.15.78:0/2953365381 wait complete. 2024-09-15T20:46:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:40.865+0000 7faace2ba640 1 librados: shutdown 2024-09-15T20:46:40.902 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:46:40.902 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:46:40.902 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:46:41.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient: get_monmap_and_config 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient: build_initial_monmap 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient: monmap: 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:41.018117+0000 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:41.018117+0000 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:41.016 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:41.017 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 1 Processor -- start 2024-09-15T20:46:41.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 1 -- start start 2024-09-15T20:46:41.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient: init 2024-09-15T20:46:41.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:41.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient(hunting): picked mon.noname-c con 0x7f9318134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient(hunting): picked mon.noname-b con 0x7f931809f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient(hunting): picked mon.noname-a con 0x7f931806e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f93180a0470 con 0x7f931806e960 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f931809feb0 con 0x7f9318134bb0 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f931809e4a0 con 0x7f931809f5a0 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient(hunting): authenticate will time out at 2061.902757s 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f9317fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f931809f5a0 0x7f93181399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53172/0 (socket says 172.21.15.78:53172) 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f9317fff640 1 -- 172.21.15.78:0/589740245 learned_addr learned my addr 172.21.15.78:0/589740245 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1946607233 0 0) 0x7f931809e4a0 con 0x7f931809f5a0 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient(hunting): my global_id is 124349 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9304003530 con 0x7f931809f5a0 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 789643261 0 0) 0x7f93080014a0 con 0x7f931806e960 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient(hunting): my global_id is 144147 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f931809e4a0 con 0x7f931806e960 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1715295789 0 0) 0x7f9304003530 con 0x7f931809f5a0 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 >> v1:172.21.15.146:6789/0 conn(0x7f9318134bb0 legacy=0x7f9318134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 >> v1:172.21.15.78:6789/0 conn(0x7f931806e960 legacy=0x7f931813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: _finish_hunting 0 2024-09-15T20:46:41.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 monclient: found mon.noname-b 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f93180a0ff0 con 0x7f931809f5a0 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: _finish_auth 0 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: _check_auth_tickets 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f93000016a0 con 0x7f931809f5a0 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 5 monclient: authenticate success, global_id 124349 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: dump: 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.027 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f93000018c0 con 0x7f931809f5a0 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 1 -- 172.21.15.78:0/589740245 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9300001d50 con 0x7f931809f5a0 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:41.028 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f931ebfb640 4 set_mon_vals no callback set 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.013+0000 7f93177fe640 10 monclient: dump: 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.029 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: shutdown 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- 172.21.15.78:0/589740245 >> v1:172.21.15.78:6790/0 conn(0x7f931809f5a0 legacy=0x7f93181399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- 172.21.15.78:0/589740245 shutdown_connections 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- 172.21.15.78:0/589740245 >> 172.21.15.78:0/589740245 conn(0x7f93180a63c0 msgr2=0x7f93180a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- 172.21.15.78:0/589740245 shutdown_connections 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- 172.21.15.78:0/589740245 wait complete. 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: build_initial_monmap 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: monmap: 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:41.030 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 librados: starting msgr at 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 librados: starting objecter 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 Processor -- start 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- start start 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 librados: setting wanted keys 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 librados: calling monclient init 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: init 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:46:41.031 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient(hunting): picked mon.noname-b con 0x7f931806e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient(hunting): picked mon.noname-a con 0x7f931809f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient(hunting): picked mon.noname-c con 0x7f9318134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9318073b30 con 0x7f931809f5a0 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f931809d180 con 0x7f931806e960 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9318122020 con 0x7f9318134bb0 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient(hunting): authenticate will time out at 2061.905942s 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f9317fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f931809f5a0 0x7f93181cd9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:53914/0 (socket says 172.21.15.78:53914) 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f9317fff640 1 -- 172.21.15.78:0/73347167 learned_addr learned my addr 172.21.15.78:0/73347167 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 client.?.objecter ms_handle_connect 0x7f931809f5a0 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 client.?.objecter ms_handle_connect 0x7f9318134bb0 2024-09-15T20:46:41.032 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2996646946 0 0) 0x7f9318122020 con 0x7f9318134bb0 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): my global_id is 124352 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92f8004ef0 con 0x7f9318134bb0 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4063715633 0 0) 0x7f9318073b30 con 0x7f931809f5a0 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): my global_id is 144150 2024-09-15T20:46:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9318122020 con 0x7f931809f5a0 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 client.?.objecter ms_handle_connect 0x7f931806e960 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 559480537 0 0) 0x7f9318122020 con 0x7f931809f5a0 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9318073b30 con 0x7f931809f5a0 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9300001c10 con 0x7f931809f5a0 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): dump: 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1422164158 0 0) 0x7f92f8004ef0 con 0x7f9318134bb0 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9318122020 con 0x7f9318134bb0 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9308001770 con 0x7f9318134bb0 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): dump: 2024-09-15T20:46:41.035 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3153193346 0 0) 0x7f931809d180 con 0x7f931806e960 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient(hunting): my global_id is 134227 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92f8004ef0 con 0x7f931806e960 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1706435721 0 0) 0x7f9318073b30 con 0x7f931809f5a0 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 >> v1:172.21.15.78:6790/0 conn(0x7f9318134bb0 legacy=0x7f93181d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 >> v1:172.21.15.146:6789/0 conn(0x7f931806e960 legacy=0x7f9318133990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient: _finish_hunting 0 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 monclient: found mon.a 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9318124380 con 0x7f931809f5a0 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient: _finish_auth 0 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient: _check_auth_tickets 2024-09-15T20:46:41.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 5 monclient: authenticate success, global_id 144150 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: _renew_subs 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9318123ae0 con 0x7f931809f5a0 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 client.144150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: _renew_subs 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9318134270 con 0x7f931809f5a0 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 librados: init done 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9300004c80 con 0x7f931809f5a0 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9300004fd0 con 0x7f931809f5a0 2024-09-15T20:46:41.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93167fc640 4 set_mon_vals no callback set 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 10 monclient: dump: 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f9300011040 con 0x7f931809f5a0 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.017+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92fc0065d0 con 0x7f92fc002870 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 client.144150.objecter ms_handle_connect 0x7f92fc002870 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 client.144150.objecter resend_mon_ops 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(624..624 src has 379..624) ==== 5913+0+0 (unknown 149201672 0 0) 0x7f930004b950 con 0x7f931809f5a0 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 client.144150.objecter ms_dispatch 0x7f9318000ba0 osd_map(624..624 src has 379..624) 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 3 client.144150.objecter handle_osd_map got epochs [624,624] > 0 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 3 client.144150.objecter handle_osd_map decoding full epoch 624 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 20 client.144150.objecter dump_active .. 0 homeless 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 507771466 0 0) 0x7f92fc0065d0 con 0x7f92fc002870 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient(hunting): my global_id is 124355 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9318073b30 con 0x7f92fc002870 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f931d198640 10 monclient: get_auth_request con 0x7f92f803e390 auth_method 0 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2674484802 0 0) 0x7f9318073b30 con 0x7f92fc002870 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92fc0065d0 con 0x7f92fc002870 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9310001790 con 0x7f92fc002870 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient: dump: 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3266178698 0 0) 0x7f92fc0065d0 con 0x7f92fc002870 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f92fc00a170 con 0x7f92fc002870 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 client.144150.objecter ms_handle_connect 0x7f92f803e390 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9310001a90 con 0x7f92fc002870 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 >> v1:172.21.15.78:6790/0 conn(0x7f92fc002870 legacy=0x7f92fc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.021+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 >> v1:172.21.15.78:6790/0 conn(0x7f92fc002870 legacy=0x7f92fc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f931ebfb640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f92fc009e60 con 0x7f92fc009120 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 client.144150.objecter ms_handle_connect 0x7f92fc009120 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 client.144150.objecter resend_mon_ops 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1175758660 0 0) 0x7f92fc009e60 con 0x7f92fc009120 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 monclient(hunting): my global_id is 124358 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f92fc0065d0 con 0x7f92fc009120 2024-09-15T20:46:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2254887651 0 0) 0x7f92fc0065d0 con 0x7f92fc009120 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f92fc009e60 con 0x7f92fc009120 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f93100037a0 con 0x7f92fc009120 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 10 monclient: dump: 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3798427776 0 0) 0x7f92fc009e60 con 0x7f92fc009120 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f92fc00c400 con 0x7f92fc009120 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.025+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4305 (unknown 0 0 3076304489) 0x7f93100037e0 con 0x7f92fc009120 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f93157fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f93157fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 >> v1:172.21.15.78:6790/0 conn(0x7f92fc009120 legacy=0x7f92fc0095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f93157fa640 1 -- 172.21.15.78:0/73347167 >> v1:172.21.15.78:6790/0 conn(0x7f92fc009120 legacy=0x7f92fc0095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:46:41.046 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 60, 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 218953, 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:46:41.047 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.048 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.049 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:46:41.050 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:41.051 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.052 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.053 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:46:41.054 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:41.055 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:46:41.056 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.057 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:41.058 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.059 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.059 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.059 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:46:41.059 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:46:41.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 10 librados: watch_flush enter 2024-09-15T20:46:41.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 10 librados: watch_flush exit 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 20 client.144150.objecter shutdown clearing up homeless session... 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 10 client.144150.objecter successfully canceled tick 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 >> v1:172.21.15.146:6812/435598642 conn(0x7f92f803e390 legacy=0x7f92f8040850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 10 monclient: shutdown 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 >> v1:172.21.15.78:6789/0 conn(0x7f931809f5a0 legacy=0x7f93181cd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931d198640 1 -- 172.21.15.78:0/73347167 reap_dead start 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931d198640 1 -- 172.21.15.78:0/73347167 reap_dead start 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 shutdown_connections 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 >> 172.21.15.78:0/73347167 conn(0x7f93180a63c0 msgr2=0x7f931809d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 shutdown_connections 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 1 -- 172.21.15.78:0/73347167 wait complete. 2024-09-15T20:46:41.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.029+0000 7f931ebfb640 1 librados: shutdown 2024-09-15T20:46:41.062 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:46:41.062 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:46:41.062 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:46:41.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient: get_monmap_and_config 2024-09-15T20:46:41.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient: build_initial_monmap 2024-09-15T20:46:41.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient: monmap: 2024-09-15T20:46:41.179 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:41.180 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.180 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:41.181185+0000 2024-09-15T20:46:41.180 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:41.181185+0000 2024-09-15T20:46:41.180 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:41.180 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.180 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:41.181 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:41.181 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:41.181 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 Processor -- start 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- start start 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient: init 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient(hunting): picked mon.noname-b con 0x7f9630135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient(hunting): picked mon.noname-a con 0x7f96300a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient(hunting): picked mon.noname-c con 0x7f963013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f96300745e0 con 0x7f96300a24e0 2024-09-15T20:46:41.195 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f963006ecf0 con 0x7f963013c220 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f963006efd0 con 0x7f9630135b60 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient(hunting): authenticate will time out at 2062.065862s 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962e59c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f9630135b60 0x7f9630135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:53196/0 (socket says 172.21.15.78:53196) 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962e59c640 1 -- 172.21.15.78:0/3736875186 learned_addr learned my addr 172.21.15.78:0/3736875186 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3361025356 0 0) 0x7f96300745e0 con 0x7f96300a24e0 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient(hunting): my global_id is 144153 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f961c003530 con 0x7f96300a24e0 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3584385532 0 0) 0x7f963006efd0 con 0x7f9630135b60 2024-09-15T20:46:41.196 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient(hunting): my global_id is 124361 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96300745e0 con 0x7f9630135b60 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1971414329 0 0) 0x7f961c003530 con 0x7f96300a24e0 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 >> v1:172.21.15.78:6790/0 conn(0x7f9630135b60 legacy=0x7f9630135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 >> v1:172.21.15.146:6789/0 conn(0x7f963013c220 legacy=0x7f963013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: _finish_hunting 0 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 monclient: found mon.noname-a 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9630136650 con 0x7f96300a24e0 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: _finish_auth 0 2024-09-15T20:46:41.197 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: _check_auth_tickets 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f96180016a0 con 0x7f96300a24e0 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 5 monclient: authenticate success, global_id 144153 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: dump: 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.198 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9618001840 con 0x7f96300a24e0 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient: get_monmap_and_config success 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 1 -- 172.21.15.78:0/3736875186 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9618001c50 con 0x7f96300a24e0 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 4 set_mon_vals no callback set 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f962d59a640 10 monclient: dump: 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.199 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 10 monclient: shutdown 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- 172.21.15.78:0/3736875186 >> v1:172.21.15.78:6789/0 conn(0x7f96300a24e0 legacy=0x7f963013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- 172.21.15.78:0/3736875186 shutdown_connections 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- 172.21.15.78:0/3736875186 >> 172.21.15.78:0/3736875186 conn(0x7f96300a63c0 msgr2=0x7f96300a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- 172.21.15.78:0/3736875186 shutdown_connections 2024-09-15T20:46:41.200 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.177+0000 7f9634c94640 1 -- 172.21.15.78:0/3736875186 wait complete. 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: build_initial_monmap 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: monmap: 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 librados: starting msgr at 2024-09-15T20:46:41.201 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 librados: starting objecter 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 Processor -- start 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 -- start start 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 librados: setting wanted keys 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 librados: calling monclient init 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: init 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient(hunting): picked mon.noname-b con 0x7f96300a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient(hunting): picked mon.noname-c con 0x7f9630135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient(hunting): picked mon.noname-a con 0x7f963013c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f963013b590 con 0x7f963013c220 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9630134fd0 con 0x7f96300a24e0 2024-09-15T20:46:41.202 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9630138670 con 0x7f9630135b60 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient(hunting): authenticate will time out at 2062.068883s 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f962ed9d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f963013c220 0x7f96301d2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:53942/0 (socket says 172.21.15.78:53942) 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f962ed9d640 1 -- 172.21.15.78:0/1306735594 learned_addr learned my addr 172.21.15.78:0/1306735594 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.?.objecter ms_handle_connect 0x7f963013c220 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.?.objecter ms_handle_connect 0x7f9630135b60 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1769289368 0 0) 0x7f9630138670 con 0x7f9630135b60 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): my global_id is 124364 2024-09-15T20:46:41.203 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9610004ec0 con 0x7f9630135b60 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2198063039 0 0) 0x7f963013b590 con 0x7f963013c220 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): my global_id is 144156 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9630138670 con 0x7f963013c220 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.?.objecter ms_handle_connect 0x7f96300a24e0 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3176794009 0 0) 0x7f9610004ec0 con 0x7f9630135b60 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f963013b590 con 0x7f9630135b60 2024-09-15T20:46:41.204 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9618001b90 con 0x7f9630135b60 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): dump: 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.205 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1059296938 0 0) 0x7f9630138670 con 0x7f963013c220 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9610004ec0 con 0x7f963013c220 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9620001900 con 0x7f963013c220 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): dump: 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.206 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.207 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.207 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1954553733 0 0) 0x7f9630134fd0 con 0x7f96300a24e0 2024-09-15T20:46:41.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): my global_id is 134233 2024-09-15T20:46:41.207 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9630138670 con 0x7f96300a24e0 2024-09-15T20:46:41.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2965552353 0 0) 0x7f963013b590 con 0x7f9630135b60 2024-09-15T20:46:41.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 >> v1:172.21.15.146:6789/0 conn(0x7f96300a24e0 legacy=0x7f9630071cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 >> v1:172.21.15.78:6789/0 conn(0x7f963013c220 legacy=0x7f96301d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:46:41.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 monclient: found mon.c 2024-09-15T20:46:41.208 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f963009d180 con 0x7f9630135b60 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient: _finish_auth 0 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 5 monclient: authenticate success, global_id 124364 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: _renew_subs 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9630073510 con 0x7f9630135b60 2024-09-15T20:46:41.209 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 client.124364.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: _renew_subs 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f963013b8d0 con 0x7f9630135b60 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 librados: init done 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9618001820 con 0x7f9630135b60 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9618003b10 con 0x7f9630135b60 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f962cd99640 4 set_mon_vals no callback set 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient: dump: 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.210 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f96140065d0 con 0x7f9614002870 2024-09-15T20:46:41.211 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f961800fb80 con 0x7f9630135b60 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(624..624 src has 379..624) ==== 5913+0+0 (unknown 149201672 0 0) 0x7f961804a720 con 0x7f9630135b60 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.124364.objecter ms_dispatch 0x7f9630000ba0 osd_map(624..624 src has 379..624) 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 3 client.124364.objecter handle_osd_map got epochs [624,624] > 0 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 3 client.124364.objecter handle_osd_map decoding full epoch 624 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 20 client.124364.objecter dump_active .. 0 homeless 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.124364.objecter ms_handle_connect 0x7f9614002870 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.124364.objecter resend_mon_ops 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f962ed9d640 10 monclient: get_auth_request con 0x7f961003e290 auth_method 0 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4285291164 0 0) 0x7f96140065d0 con 0x7f9614002870 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.212 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 monclient(hunting): my global_id is 134236 2024-09-15T20:46:41.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f963013b590 con 0x7f9614002870 2024-09-15T20:46:41.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.181+0000 7f9626ffd640 10 client.124364.objecter ms_handle_connect 0x7f961003e290 2024-09-15T20:46:41.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4117310775 0 0) 0x7f963013b590 con 0x7f9614002870 2024-09-15T20:46:41.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f96140065d0 con 0x7f9614002870 2024-09-15T20:46:41.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f96280015a0 con 0x7f9614002870 2024-09-15T20:46:41.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.213 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 10 monclient: dump: 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1748971058 0 0) 0x7f96140065d0 con 0x7f9614002870 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f961400a170 con 0x7f9614002870 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9628001630 con 0x7f9614002870 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 >> v1:172.21.15.146:6789/0 conn(0x7f9614002870 legacy=0x7f9614004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.185+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 >> v1:172.21.15.146:6789/0 conn(0x7f9614002870 legacy=0x7f9614004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9634c94640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9614009e80 con 0x7f9614009140 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9626ffd640 10 client.124364.objecter ms_handle_connect 0x7f9614009140 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9626ffd640 10 client.124364.objecter resend_mon_ops 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1055005145 0 0) 0x7f9614009e80 con 0x7f9614009140 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9626ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9626ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9626ffd640 10 monclient(hunting): my global_id is 134239 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.189+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f96140065d0 con 0x7f9614009140 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3481210136 0 0) 0x7f96140065d0 con 0x7f9614009140 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9614009e80 con 0x7f9614009140 2024-09-15T20:46:41.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9628004d10 con 0x7f9614009140 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 10 monclient: dump: 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:46:41.217 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:46:41.218 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2889823897 0 0) 0x7f9614009e80 con 0x7f9614009140 2024-09-15T20:46:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f961400c400 con 0x7f9614009140 2024-09-15T20:46:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 3615226308) 0x7f9628006200 con 0x7f9614009140 2024-09-15T20:46:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:46:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:46:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 >> v1:172.21.15.146:6789/0 conn(0x7f9614009140 legacy=0x7f96140095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9626ffd640 1 -- 172.21.15.78:0/1306735594 >> v1:172.21.15.146:6789/0 conn(0x7f9614009140 legacy=0x7f96140095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 60, 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 111066, 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:46:41.219 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:41.220 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:41.221 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:46:41.222 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:46:41.223 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:46:41.223 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:46:41.223 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:46:41.223 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:41.223 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:46:41.224 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:46:41.224 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:46:41.224 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:46:41.224 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:46:41.224 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:46:41.224 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:46:41.224 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:46:41.225 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:46:41.225 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:46:41.225 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:46:41.225 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:46:41.225 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:46:41.225 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.226 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.227 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:41.228 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.229 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:46:41.230 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:41.231 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:46:41.232 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:46:41.233 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:46:41.233 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:46:41.233 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:46:41.233 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:46:41.233 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 10 librados: watch_flush enter 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 10 librados: watch_flush exit 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 20 client.124364.objecter shutdown clearing up homeless session... 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 10 client.124364.objecter successfully canceled tick 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 >> v1:172.21.15.146:6812/435598642 conn(0x7f961003e290 legacy=0x7f9610040750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 10 monclient: shutdown 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 >> v1:172.21.15.78:6790/0 conn(0x7f9630135b60 legacy=0x7f96300723c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f962ed9d640 1 -- 172.21.15.78:0/1306735594 reap_dead start 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f962ed9d640 1 -- 172.21.15.78:0/1306735594 reap_dead start 2024-09-15T20:46:41.235 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 shutdown_connections 2024-09-15T20:46:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 >> 172.21.15.78:0/1306735594 conn(0x7f96300a63c0 msgr2=0x7f963012ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:46:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 shutdown_connections 2024-09-15T20:46:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 1 -- 172.21.15.78:0/1306735594 wait complete. 2024-09-15T20:46:41.236 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:41.193+0000 7f9634c94640 1 librados: shutdown 2024-09-15T20:46:41.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:46:41.237 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-09-15T20:46:41.237 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['b'] 2024-09-15T20:46:41.237 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-09-15T20:46:41.237 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-09-15T20:46:41.237 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T20:46:41.237 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:46:41.250 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T20:46:41.251 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T20:46:41.251 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T20:46:41.251 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T20:46:56.252 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T20:46:56.253 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T20:46:56.253 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:46:56.253 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:46:56.387 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient: get_monmap_and_config 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient: build_initial_monmap 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient: monmap: 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:46:56.390047+0000 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:46:56.390047+0000 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:46:56.388 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:46:56.389 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:46:56.389 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 1 Processor -- start 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 1 -- start start 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient: init 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient: _reopen_session rank -1 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient(hunting): picked mon.noname-a con 0x7f7c88136d20 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient(hunting): picked mon.noname-b con 0x7f7c88074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient(hunting): picked mon.noname-c con 0x7f7c880a5b90 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:56.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c88073560 con 0x7f7c88136d20 2024-09-15T20:46:56.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c88073840 con 0x7f7c880a5b90 2024-09-15T20:46:56.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c88073b20 con 0x7f7c88074500 2024-09-15T20:46:56.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:56.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c8ddd6640 10 monclient(hunting): authenticate will time out at 2077.274895s 2024-09-15T20:46:56.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c86ffd640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c8813cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:46:56.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c877fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7c88136d20 0x7f7c88137100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34680/0 (socket says 172.21.15.78:34680) 2024-09-15T20:46:56.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.388+0000 7f7c877fe640 1 -- 172.21.15.78:0/2239464173 learned_addr learned my addr 172.21.15.78:0/2239464173 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:46:56.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.592+0000 7f7c86ffd640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c8813cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:46:56.996 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:56.996+0000 7f7c86ffd640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c8813cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:46:57.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:57.800+0000 7f7c86ffd640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c8813cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 10 monclient(hunting): tick 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 1 monclient(hunting): continuing hunt 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c8813cc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.146:6789/0 conn(0x7f7c880a5b90 legacy=0x7f7c881403c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6789/0 conn(0x7f7c88136d20 legacy=0x7f7c88137100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 10 monclient(hunting): picked mon.noname-c con 0x7f7c740036d0 addr v1:172.21.15.146:6789/0 2024-09-15T20:46:59.390 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 10 monclient(hunting): picked mon.noname-a con 0x7f7c740072f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:46:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 10 monclient(hunting): picked mon.noname-b con 0x7f7c88074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:46:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 1 -- 172.21.15.78:0/2239464173 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c8809cfe0 con 0x7f7c740072f0 2024-09-15T20:46:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 1 -- 172.21.15.78:0/2239464173 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c88073b20 con 0x7f7c740036d0 2024-09-15T20:46:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 1 -- 172.21.15.78:0/2239464173 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c74010940 con 0x7f7c88074500 2024-09-15T20:46:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c857fa640 10 monclient(hunting): _renew_subs 2024-09-15T20:46:59.391 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.388+0000 7f7c87fff640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c7400d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:46:59.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.592+0000 7f7c87fff640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c7400d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:46:59.995 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:46:59.996+0000 7f7c87fff640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c7400d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:47:00.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:00.800+0000 7f7c87fff640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c7400d0b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:47:01.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 343238080 0 0) 0x7f7c88073b20 con 0x7f7c740036d0 2024-09-15T20:47:01.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient(hunting): my global_id is 134242 2024-09-15T20:47:01.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c70003530 con 0x7f7c740036d0 2024-09-15T20:47:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3376490445 0 0) 0x7f7c70003530 con 0x7f7c740036d0 2024-09-15T20:47:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6790/0 conn(0x7f7c88074500 legacy=0x7f7c7400d0b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:47:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.78:6789/0 conn(0x7f7c740072f0 legacy=0x7f7c74009780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: _finish_hunting 0 2024-09-15T20:47:01.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 monclient: found mon.noname-c 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.146:6789/0 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c74011bc0 con 0x7f7c740036d0 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: _finish_auth 0 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: _check_auth_tickets 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c8ddd6640 5 monclient: authenticate success, global_id 134242 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7c780015a0 con 0x7f7c740036d0 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c8ddd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: dump: 2024-09-15T20:47:01.308 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.309 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.309 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.309 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.309 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c8ddd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 <== mon.1 v1:172.21.15.146:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c78001740 con 0x7f7c740036d0 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c8ddd6640 10 monclient: get_monmap_and_config success 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c8ddd6640 4 set_mon_vals no callback set 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 1 -- 172.21.15.78:0/2239464173 <== mon.1 v1:172.21.15.146:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7c78001b50 con 0x7f7c740036d0 2024-09-15T20:47:01.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.304+0000 7f7c867fc640 10 monclient: dump: 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient: shutdown 2024-09-15T20:47:01.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:47:01.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/2239464173 >> v1:172.21.15.146:6789/0 conn(0x7f7c740036d0 legacy=0x7f7c74005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/2239464173 shutdown_connections 2024-09-15T20:47:01.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/2239464173 >> 172.21.15.78:0/2239464173 conn(0x7f7c880aa540 msgr2=0x7f7c880aa950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:47:01.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/2239464173 shutdown_connections 2024-09-15T20:47:01.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/2239464173 wait complete. 2024-09-15T20:47:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient: build_initial_monmap 2024-09-15T20:47:01.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient: monmap: 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 librados: starting msgr at 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 librados: starting objecter 2024-09-15T20:47:01.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 Processor -- start 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- start start 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 librados: setting wanted keys 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 librados: calling monclient init 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient: init 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient: _reopen_session rank -1 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient(hunting): picked mon.noname-a con 0x7f7c740036d0 addr v1:172.21.15.78:6789/0 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient(hunting): picked mon.noname-c con 0x7f7c740072f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient(hunting): picked mon.noname-b con 0x7f7c88074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c881285a0 con 0x7f7c740036d0 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c880a1330 con 0x7f7c88074500 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7c88127260 con 0x7f7c740072f0 2024-09-15T20:47:01.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient(hunting): _renew_subs 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c8ddd6640 10 monclient(hunting): authenticate will time out at 2082.195018s 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c86ffd640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f7c740072f0 legacy=0x7f7c881d1bb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c877fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7c740036d0 0x7f7c88135cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44852/0 (socket says 172.21.15.78:44852) 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c877fe640 1 -- 172.21.15.78:0/1873246976 learned_addr learned my addr 172.21.15.78:0/1873246976 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 10 client.?.objecter ms_handle_connect 0x7f7c740036d0 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 10 client.?.objecter ms_handle_connect 0x7f7c88074500 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2655241986 0 0) 0x7f7c880a1330 con 0x7f7c88074500 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 10 monclient(hunting): my global_id is 134245 2024-09-15T20:47:01.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.308+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7c74006a10 con 0x7f7c88074500 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3342598670 0 0) 0x7f7c74006a10 con 0x7f7c88074500 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7c880a1330 con 0x7f7c88074500 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7c80001be0 con 0x7f7c88074500 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient(hunting): dump: 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.319 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 207836510 0 0) 0x7f7c880a1330 con 0x7f7c88074500 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 >> v1:172.21.15.78:6790/0 conn(0x7f7c740072f0 legacy=0x7f7c881d1bb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 >> v1:172.21.15.78:6789/0 conn(0x7f7c740036d0 legacy=0x7f7c88135cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 monclient: found mon.b 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 --> v1:172.21.15.146:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c88126240 con 0x7f7c88074500 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient: _finish_auth 0 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:47:01.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 5 monclient: authenticate success, global_id 134245 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 10 monclient: _renew_subs 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 --> v1:172.21.15.146:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7c881282c0 con 0x7f7c88074500 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 10 client.134245.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 10 monclient: _renew_subs 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 --> v1:172.21.15.146:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7c88136400 con 0x7f7c88074500 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 1 librados: init done 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7c800039a0 con 0x7f7c88074500 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7c80003db0 con 0x7f7c88074500 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:47:01.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c85ffb640 4 set_mon_vals no callback set 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 monclient: dump: 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:47:01.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c68005090 con 0x7f7c88074500 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f7c80003fd0 con 0x7f7c88074500 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 8 ==== osd_map(625..625 src has 379..625) ==== 5913+0+0 (unknown 321245661 0 0) 0x7f7c8004bc60 con 0x7f7c88074500 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 10 client.134245.objecter ms_dispatch 0x7f7c88000ba0 osd_map(625..625 src has 379..625) 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 3 client.134245.objecter handle_osd_map got epochs [625,625] > 0 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.312+0000 7f7c6ffff640 3 client.134245.objecter handle_osd_map decoding full epoch 625 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.316+0000 7f7c6ffff640 20 client.134245.objecter dump_active .. 0 homeless 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.316+0000 7f7c877fe640 10 monclient: get_auth_request con 0x7f7c7403ecf0 auth_method 0 2024-09-15T20:47:01.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.316+0000 7f7c6ffff640 10 client.134245.objecter ms_handle_connect 0x7f7c7403ecf0 2024-09-15T20:47:01.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.332+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f7c80016d30 con 0x7f7c88074500 2024-09-15T20:47:01.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.332+0000 7f7c6ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:47:01.331 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.332+0000 7f7c6ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:47:01.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.568+0000 7f7c8ddd6640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:47:01.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.568+0000 7f7c8ddd6640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:47:01.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.568+0000 7f7c8ddd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.146:6789/0 2024-09-15T20:47:01.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.568+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 --> v1:172.21.15.146:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7c68005380 con 0x7f7c88074500 2024-09-15T20:47:01.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.568+0000 7f7c6ffff640 1 -- 172.21.15.78:0/1873246976 <== mon.1 v1:172.21.15.146:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 3640464179) 0x7f7c80016630 con 0x7f7c88074500 2024-09-15T20:47:01.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.568+0000 7f7c6ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:47:01.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.568+0000 7f7c6ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:47:01.569 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":62,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:47:01.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 10 librados: watch_flush enter 2024-09-15T20:47:01.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 10 librados: watch_flush exit 2024-09-15T20:47:01.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 20 client.134245.objecter shutdown clearing up homeless session... 2024-09-15T20:47:01.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 10 client.134245.objecter successfully canceled tick 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 >> v1:172.21.15.146:6812/435598642 conn(0x7f7c7403ecf0 legacy=0x7f7c740411b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 10 monclient: shutdown 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 >> v1:172.21.15.146:6789/0 conn(0x7f7c88074500 legacy=0x7f7c881d52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 shutdown_connections 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 >> 172.21.15.78:0/1873246976 conn(0x7f7c880aa540 msgr2=0x7f7c880a2fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 shutdown_connections 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 1 -- 172.21.15.78:0/1873246976 wait complete. 2024-09-15T20:47:01.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.572+0000 7f7c8ddd6640 1 librados: shutdown 2024-09-15T20:47:01.588 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:47:01.588 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T20:47:01.588 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:47:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.708+0000 7f2f90589640 10 monclient: get_monmap_and_config 2024-09-15T20:47:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.708+0000 7f2f90589640 10 monclient: build_initial_monmap 2024-09-15T20:47:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.708+0000 7f2f90589640 10 monclient: monmap: 2024-09-15T20:47:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:47:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:47:01.711486+0000 2024-09-15T20:47:01.709 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:47:01.711486+0000 2024-09-15T20:47:01.710 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:47:01.710 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.710 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:47:01.710 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:47:01.710 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:47:01.710 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.708+0000 7f2f90589640 1 Processor -- start 2024-09-15T20:47:01.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.708+0000 7f2f90589640 1 -- start start 2024-09-15T20:47:01.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.708+0000 7f2f90589640 10 monclient: init 2024-09-15T20:47:01.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: _reopen_session rank -1 2024-09-15T20:47:01.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:47:01.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): picked mon.noname-a con 0x7f2f88137d60 addr v1:172.21.15.78:6789/0 2024-09-15T20:47:01.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): picked mon.noname-b con 0x7f2f88074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:47:01.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): picked mon.noname-c con 0x7f2f8813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f88138850 con 0x7f2f88137d60 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f880a5da0 con 0x7f2f8813e370 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f88073560 con 0x7f2f88074500 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): _renew_subs 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): authenticate will time out at 2082.596356s 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8db24640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f2f88074500 legacy=0x7f2f8813cc50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8e325640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2f88137d60 0x7f2f88138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44862/0 (socket says 172.21.15.78:44862) 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8e325640 1 -- 172.21.15.78:0/3344851183 learned_addr learned my addr 172.21.15.78:0/3344851183 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 325383432 0 0) 0x7f2f88138850 con 0x7f2f88137d60 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient(hunting): my global_id is 144159 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f74003530 con 0x7f2f88137d60 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2206160310 0 0) 0x7f2f74003530 con 0x7f2f88137d60 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 >> v1:172.21.15.78:6790/0 conn(0x7f2f88074500 legacy=0x7f2f8813cc50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 >> v1:172.21.15.146:6789/0 conn(0x7f2f8813e370 legacy=0x7f2f88140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: _finish_hunting 0 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 monclient: found mon.noname-a 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f88073840 con 0x7f2f88137d60 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: _finish_auth 0 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: _check_auth_tickets 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:47:01.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2f840016a0 con 0x7f2f88137d60 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: dump: 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 5 monclient: authenticate success, global_id 144159 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f840018c0 con 0x7f2f88137d60 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: handle_config config(0 keys) 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: get_monmap_and_config success 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 4 set_mon_vals no callback set 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 1 -- 172.21.15.78:0/3344851183 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2f84001cd0 con 0x7f2f88137d60 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8d323640 10 monclient: dump: 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.723 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: shutdown 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- 172.21.15.78:0/3344851183 >> v1:172.21.15.78:6789/0 conn(0x7f2f88137d60 legacy=0x7f2f88138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- 172.21.15.78:0/3344851183 shutdown_connections 2024-09-15T20:47:01.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- 172.21.15.78:0/3344851183 >> 172.21.15.78:0/3344851183 conn(0x7f2f880aa650 msgr2=0x7f2f880aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:47:01.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- 172.21.15.78:0/3344851183 shutdown_connections 2024-09-15T20:47:01.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- 172.21.15.78:0/3344851183 wait complete. 2024-09-15T20:47:01.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: build_initial_monmap 2024-09-15T20:47:01.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: monmap: 2024-09-15T20:47:01.725 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:47:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:47:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:47:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:47:01.726 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 librados: starting msgr at 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 librados: starting objecter 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 Processor -- start 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- start start 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 librados: setting wanted keys 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 librados: calling monclient init 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: init 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: _reopen_session rank -1 2024-09-15T20:47:01.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:47:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): picked mon.noname-a con 0x7f2f88074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:47:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): picked mon.noname-c con 0x7f2f88137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:47:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): picked mon.noname-b con 0x7f2f8813e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:47:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f88132ba0 con 0x7f2f88074500 2024-09-15T20:47:01.728 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f88132e80 con 0x7f2f8813e370 2024-09-15T20:47:01.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f880a3ef0 con 0x7f2f88137d60 2024-09-15T20:47:01.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): _renew_subs 2024-09-15T20:47:01.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8db24640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f2f88137d60 legacy=0x7f2f8813f2c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:47:01.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f90589640 10 monclient(hunting): authenticate will time out at 2082.599627s 2024-09-15T20:47:01.730 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8e325640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f2f88074500 0x7f2f8813ebb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44866/0 (socket says 172.21.15.78:44866) 2024-09-15T20:47:01.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f8e325640 1 -- 172.21.15.78:0/3871336977 learned_addr learned my addr 172.21.15.78:0/3871336977 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:47:01.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f7effd640 10 client.?.objecter ms_handle_connect 0x7f2f88074500 2024-09-15T20:47:01.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f7effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:47:01.731 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 914774307 0 0) 0x7f2f88132ba0 con 0x7f2f88074500 2024-09-15T20:47:01.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f7effd640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f7effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f7effd640 10 monclient(hunting): my global_id is 144162 2024-09-15T20:47:01.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.712+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f6c004ec0 con 0x7f2f88074500 2024-09-15T20:47:01.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 client.?.objecter ms_handle_connect 0x7f2f8813e370 2024-09-15T20:47:01.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:47:01.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4286451707 0 0) 0x7f2f6c004ec0 con 0x7f2f88074500 2024-09-15T20:47:01.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f88132ba0 con 0x7f2f88074500 2024-09-15T20:47:01.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2f84001c10 con 0x7f2f88074500 2024-09-15T20:47:01.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:47:01.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:47:01.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): dump: 2024-09-15T20:47:01.734 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.734 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.734 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.734 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.734 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.734 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.734 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.734 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.735 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.735 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:47:01.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1204565952 0 0) 0x7f2f88132e80 con 0x7f2f8813e370 2024-09-15T20:47:01.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): my global_id is 134251 2024-09-15T20:47:01.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f6c004ec0 con 0x7f2f8813e370 2024-09-15T20:47:01.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1209894550 0 0) 0x7f2f88132ba0 con 0x7f2f88074500 2024-09-15T20:47:01.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 >> v1:172.21.15.78:6790/0 conn(0x7f2f88137d60 legacy=0x7f2f8813f2c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:47:01.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 >> v1:172.21.15.146:6789/0 conn(0x7f2f8813e370 legacy=0x7f2f8813f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: _finish_hunting 0 2024-09-15T20:47:01.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 monclient: found mon.a 2024-09-15T20:47:01.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:47:01.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:47:01.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f88141910 con 0x7f2f88074500 2024-09-15T20:47:01.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: _finish_auth 0 2024-09-15T20:47:01.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: _check_auth_tickets 2024-09-15T20:47:01.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:47:01.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 5 monclient: authenticate success, global_id 144162 2024-09-15T20:47:01.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 10 monclient: _renew_subs 2024-09-15T20:47:01.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:47:01.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f2f8809e0b0 con 0x7f2f88074500 2024-09-15T20:47:01.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 10 client.144162.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:47:01.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 10 monclient: _renew_subs 2024-09-15T20:47:01.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:47:01.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f2f881d5e10 con 0x7f2f88074500 2024-09-15T20:47:01.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 1 librados: init done 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f2f84004c00 con 0x7f2f88074500 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2f84004f50 con 0x7f2f88074500 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f8cb22640 4 set_mon_vals no callback set 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: dump: 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.740 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2f700065d0 con 0x7f2f70002870 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 client.144162.objecter ms_handle_connect 0x7f2f70002870 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 client.144162.objecter resend_mon_ops 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3588871115 0 0) 0x7f2f700065d0 con 0x7f2f70002870 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient(hunting): my global_id is 144165 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f88132ba0 con 0x7f2f70002870 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 956023794 0 0) 0x7f2f88132ba0 con 0x7f2f70002870 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f700065d0 con 0x7f2f70002870 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2f78003fb0 con 0x7f2f70002870 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: dump: 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.742 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.743 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.743 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.743 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f2f84010fc0 con 0x7f2f88074500 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(625..625 src has 379..625) ==== 5913+0+0 (unknown 321245661 0 0) 0x7f2f8404b9c0 con 0x7f2f88074500 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 client.144162.objecter ms_dispatch 0x7f2f88000ba0 osd_map(625..625 src has 379..625) 2024-09-15T20:47:01.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 3 client.144162.objecter handle_osd_map got epochs [625,625] > 0 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 3 client.144162.objecter handle_osd_map decoding full epoch 625 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 20 client.144162.objecter dump_active .. 0 homeless 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 228471734 0 0) 0x7f2f700065d0 con 0x7f2f70002870 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2f7000a170 con 0x7f2f70002870 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f8eb26640 10 monclient: get_auth_request con 0x7f2f6c03e1f0 auth_method 0 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f2f78006270 con 0x7f2f70002870 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 >> v1:172.21.15.78:6789/0 conn(0x7f2f70002870 legacy=0x7f2f70004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 >> v1:172.21.15.78:6789/0 conn(0x7f2f70002870 legacy=0x7f2f70004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:47:01.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.716+0000 7f2f7effd640 10 client.144162.objecter ms_handle_connect 0x7f2f6c03e1f0 2024-09-15T20:47:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:47:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f2f70009e90 con 0x7f2f70009150 2024-09-15T20:47:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 client.144162.objecter ms_handle_connect 0x7f2f70009150 2024-09-15T20:47:01.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 client.144162.objecter resend_mon_ops 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614629161 0 0) 0x7f2f70009e90 con 0x7f2f70009150 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient(hunting): my global_id is 144168 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2f700065d0 con 0x7f2f70009150 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2768674298 0 0) 0x7f2f700065d0 con 0x7f2f70009150 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f2f70009e90 con 0x7f2f70009150 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f2f78004980 con 0x7f2f70009150 2024-09-15T20:47:01.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:47:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient: dump: 2024-09-15T20:47:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.749 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 322261445 0 0) 0x7f2f70009e90 con 0x7f2f70009150 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2f7000c420 con 0x7f2f70009150 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4295 (unknown 0 0 350068715) 0x7f2f78005cb0 con 0x7f2f70009150 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 >> v1:172.21.15.78:6789/0 conn(0x7f2f70009150 legacy=0x7f2f700095e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f7effd640 1 -- 172.21.15.78:0/3871336977 >> v1:172.21.15.78:6789/0 conn(0x7f2f70009150 legacy=0x7f2f700095e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 62, 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 23619, 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:47:01.751 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-09-15T20:47:01.753 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.753 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:47:01.753 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:47:01.753 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:47:01.753 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:47:01.753 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:47:01.754 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:47:01.756 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:47:01.757 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:47:01.758 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.759 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:47:01.760 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:47:01.761 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.762 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:47:01.763 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:47:01.764 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:47:01.765 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.765 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.765 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.765 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:47:01.765 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 10 librados: watch_flush enter 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 10 librados: watch_flush exit 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 20 client.144162.objecter shutdown clearing up homeless session... 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 10 client.144162.objecter successfully canceled tick 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 >> v1:172.21.15.146:6812/435598642 conn(0x7f2f6c03e1f0 legacy=0x7f2f6c0406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 10 monclient: shutdown 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 >> v1:172.21.15.78:6789/0 conn(0x7f2f88074500 legacy=0x7f2f8813ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f8eb26640 1 -- 172.21.15.78:0/3871336977 reap_dead start 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f8eb26640 1 -- 172.21.15.78:0/3871336977 reap_dead start 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 shutdown_connections 2024-09-15T20:47:01.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 >> 172.21.15.78:0/3871336977 conn(0x7f2f880aa650 msgr2=0x7f2f880a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:47:01.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 shutdown_connections 2024-09-15T20:47:01.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 1 -- 172.21.15.78:0/3871336977 wait complete. 2024-09-15T20:47:01.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.724+0000 7f2f90589640 1 librados: shutdown 2024-09-15T20:47:01.768 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:47:01.768 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-09-15T20:47:01.768 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:47:01.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.876+0000 7f543130f640 10 monclient: get_monmap_and_config 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.876+0000 7f543130f640 10 monclient: build_initial_monmap 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.876+0000 7f543130f640 10 monclient: monmap: 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:47:01.880113+0000 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:47:01.880113+0000 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:47:01.878 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 Processor -- start 2024-09-15T20:47:01.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- start start 2024-09-15T20:47:01.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: init 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: _reopen_session rank -1 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): picked mon.noname-b con 0x7f542c135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): picked mon.noname-c con 0x7f542c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): picked mon.noname-a con 0x7f542c13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c0745e0 con 0x7f542c13c220 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f542affd640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f542c135b60 legacy=0x7f542c135f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c06ecf0 con 0x7f542c0a24e0 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c06efd0 con 0x7f542c135b60 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): _renew_subs 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): authenticate will time out at 2082.764931s 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f542b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f542c13c220 0x7f542c13e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44900/0 (socket says 172.21.15.78:44900) 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f542b7fe640 1 -- 172.21.15.78:0/3072292398 learned_addr learned my addr 172.21.15.78:0/3072292398 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:47:01.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 680081192 0 0) 0x7f542c0745e0 con 0x7f542c13c220 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient(hunting): my global_id is 144171 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5414003530 con 0x7f542c13c220 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3544434608 0 0) 0x7f5414003530 con 0x7f542c13c220 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 >> v1:172.21.15.78:6790/0 conn(0x7f542c135b60 legacy=0x7f542c135f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 >> v1:172.21.15.146:6789/0 conn(0x7f542c0a24e0 legacy=0x7f542c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:47:01.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 monclient: found mon.noname-a 2024-09-15T20:47:01.889 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c136650 con 0x7f542c13c220 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: _finish_auth 0 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f541c0016a0 con 0x7f542c13c220 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 5 monclient: authenticate success, global_id 144171 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: dump: 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.890 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f541c0018c0 con 0x7f542c13c220 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 1 -- 172.21.15.78:0/3072292398 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f541c001cd0 con 0x7f542c13c220 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f5429ffb640 10 monclient: dump: 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: get_monmap_and_config success 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 4 set_mon_vals no callback set 2024-09-15T20:47:01.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: shutdown 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- 172.21.15.78:0/3072292398 >> v1:172.21.15.78:6789/0 conn(0x7f542c13c220 legacy=0x7f542c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- 172.21.15.78:0/3072292398 shutdown_connections 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- 172.21.15.78:0/3072292398 >> 172.21.15.78:0/3072292398 conn(0x7f542c0a63c0 msgr2=0x7f542c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- 172.21.15.78:0/3072292398 shutdown_connections 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- 172.21.15.78:0/3072292398 wait complete. 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: build_initial_monmap 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: monmap: 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:47:01.893 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 librados: starting msgr at 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 librados: starting objecter 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 Processor -- start 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- start start 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 librados: setting wanted keys 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 librados: calling monclient init 2024-09-15T20:47:01.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: init 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: _reopen_session rank -1 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): picked mon.noname-b con 0x7f542c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): picked mon.noname-c con 0x7f542c135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): picked mon.noname-a con 0x7f542c13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c072f90 con 0x7f542c13c220 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c072750 con 0x7f542c0a24e0 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f542c13bbd0 con 0x7f542c135b60 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): _renew_subs 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f543130f640 10 monclient(hunting): authenticate will time out at 2082.767714s 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f542a7fc640 1 -- >> v1:172.21.15.78:6790/0 conn(0x7f542c135b60 legacy=0x7f542c072040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.78:6790/0 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f542b7fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f542c13c220 0x7f542c1d4c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:44904/0 (socket says 172.21.15.78:44904) 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f542b7fe640 1 -- 172.21.15.78:0/2150329698 learned_addr learned my addr 172.21.15.78:0/2150329698 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:47:01.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f54137fe640 10 client.?.objecter ms_handle_connect 0x7f542c13c220 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f54137fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4209207128 0 0) 0x7f542c072f90 con 0x7f542c13c220 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f54137fe640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f54137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f54137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f54137fe640 10 monclient(hunting): my global_id is 144174 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.880+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5418004ef0 con 0x7f542c13c220 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3902839103 0 0) 0x7f5418004ef0 con 0x7f542c13c220 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f542c072f90 con 0x7f542c13c220 2024-09-15T20:47:01.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f541c001c10 con 0x7f542c13c220 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient(hunting): dump: 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 client.?.objecter ms_handle_connect 0x7f542c0a24e0 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2535736963 0 0) 0x7f542c072f90 con 0x7f542c13c220 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 >> v1:172.21.15.78:6790/0 conn(0x7f542c135b60 legacy=0x7f542c072040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 >> v1:172.21.15.146:6789/0 conn(0x7f542c0a24e0 legacy=0x7f542c071930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: _finish_hunting 0 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 monclient: found mon.a 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f542c13fa80 con 0x7f542c13c220 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: _finish_auth 0 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: _check_auth_tickets 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:47:01.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 5 monclient: authenticate success, global_id 144174 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 10 monclient: _renew_subs 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f542c09d180 con 0x7f542c13c220 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 10 client.144174.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 10 monclient: _renew_subs 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f542c135470 con 0x7f542c13c220 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 1 librados: init done 2024-09-15T20:47:01.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f542c09e5b0 con 0x7f542c13cc20 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f541c001860 con 0x7f542c13c220 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f541c004c00 con 0x7f542c13c220 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: dump: 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.900 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54297fa640 4 set_mon_vals no callback set 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 client.144174.objecter ms_handle_connect 0x7f542c13cc20 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 client.144174.objecter resend_mon_ops 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f541c010cb0 con 0x7f542c13c220 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(625..625 src has 379..625) ==== 5913+0+0 (unknown 321245661 0 0) 0x7f541c04ba40 con 0x7f542c13c220 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 client.144174.objecter ms_dispatch 0x7f542c000ba0 osd_map(625..625 src has 379..625) 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 3 client.144174.objecter handle_osd_map got epochs [625,625] > 0 2024-09-15T20:47:01.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 3 client.144174.objecter handle_osd_map decoding full epoch 625 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 20 client.144174.objecter dump_active .. 0 homeless 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3259759225 0 0) 0x7f542c09e5b0 con 0x7f542c13cc20 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient(hunting): my global_id is 134260 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f542c072f90 con 0x7f542c13cc20 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f542a7fc640 10 monclient: get_auth_request con 0x7f541803df20 auth_method 0 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 client.144174.objecter ms_handle_connect 0x7f541803df20 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1692340350 0 0) 0x7f542c072f90 con 0x7f542c13cc20 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f542c09e5b0 con 0x7f542c13cc20 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5420003fa0 con 0x7f542c13cc20 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:47:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: dump: 2024-09-15T20:47:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.903 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.904 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3655692842 0 0) 0x7f542c09e5b0 con 0x7f542c13cc20 2024-09-15T20:47:01.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f542c0008e0 con 0x7f542c13cc20 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f54200062e0 con 0x7f542c13cc20 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 >> v1:172.21.15.146:6789/0 conn(0x7f542c13cc20 legacy=0x7f542c13d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.884+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 >> v1:172.21.15.146:6789/0 conn(0x7f542c13cc20 legacy=0x7f542c13d000 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f543130f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f542c13daf0 con 0x7f542c13d710 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 client.144174.objecter ms_handle_connect 0x7f542c13d710 2024-09-15T20:47:01.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 client.144174.objecter resend_mon_ops 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3565072677 0 0) 0x7f542c13daf0 con 0x7f542c13d710 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient(hunting): _negotiate 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient(hunting): my global_id is 134263 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f542c09e5b0 con 0x7f542c13d710 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3611782991 0 0) 0x7f542c09e5b0 con 0x7f542c13d710 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f542c13daf0 con 0x7f542c13d710 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5420004690 con 0x7f542c13d710 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient: dump: 2024-09-15T20:47:01.907 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2676870806 0 0) 0x7f542c13daf0 con 0x7f542c13d710 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f542c1357e0 con 0x7f542c13d710 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4127 (unknown 0 0 2040296198) 0x7f5420006490 con 0x7f542c13d710 2024-09-15T20:47:01.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:47:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:47:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 >> v1:172.21.15.146:6789/0 conn(0x7f542c13d710 legacy=0x7f542c1da370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.892+0000 7f54137fe640 1 -- 172.21.15.78:0/2150329698 >> v1:172.21.15.146:6789/0 conn(0x7f542c13d710 legacy=0x7f542c1da370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:47:01.909 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:47:01.909 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:47:01.909 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:47:01.909 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 62, 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 131761, 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: 1 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:47:01.910 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:47:01.911 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.912 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:47:01.913 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:47:01.913 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:47:01.913 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:47:01.913 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:47:01.913 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:47:01.913 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:47:01.913 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:47:01.913 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:47:01.914 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:47:01.915 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:47:01.916 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:47:01.916 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:47:01.916 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.916 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:47:01.916 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:47:01.916 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:47:01.916 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.916 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:47:01.917 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:47:01.918 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:47:01.918 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:47:01.918 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.918 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:47:01.918 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:47:01.919 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.920 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.921 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:47:01.922 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:47:01.923 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:47:01.923 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:47:01.923 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:47:01.923 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:47:01.923 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:47:01.923 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:47:01.923 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:47:01.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 10 librados: watch_flush enter 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 10 librados: watch_flush exit 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 20 client.144174.objecter shutdown clearing up homeless session... 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 10 client.144174.objecter successfully canceled tick 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 >> v1:172.21.15.146:6812/435598642 conn(0x7f541803df20 legacy=0x7f54180403e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 10 monclient: shutdown 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 >> v1:172.21.15.78:6789/0 conn(0x7f542c13c220 legacy=0x7f542c1d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f542b7fe640 1 -- 172.21.15.78:0/2150329698 reap_dead start 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 shutdown_connections 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 >> 172.21.15.78:0/2150329698 conn(0x7f542c0a63c0 msgr2=0x7f542c12ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 shutdown_connections 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 1 -- 172.21.15.78:0/2150329698 wait complete. 2024-09-15T20:47:01.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:47:01.896+0000 7f543130f640 1 librados: shutdown 2024-09-15T20:47:01.927 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:47:01.927 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:48:31.927 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-09-15T20:48:31.928 INFO:tasks.ceph.mon.c:Restarting daemon 2024-09-15T20:48:31.928 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-09-15T20:48:31.931 INFO:tasks.ceph.mon.c:Started 2024-09-15T20:48:31.932 INFO:tasks.mon_thrash:Sending STOP to mon b 2024-09-15T20:48:31.932 INFO:tasks.ceph.mon.b:Sent signal 19 2024-09-15T20:48:31.932 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-09-15T20:48:46.933 INFO:tasks.mon_thrash:Sending CONT to mon b 2024-09-15T20:48:46.933 INFO:tasks.ceph.mon.b:Sent signal 18 2024-09-15T20:48:46.934 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:48:46.934 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:48:47.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.063+0000 7f4b1d6e4640 10 monclient: get_monmap_and_config 2024-09-15T20:48:47.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.063+0000 7f4b1d6e4640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.063+0000 7f4b1d6e4640 10 monclient: monmap: 2024-09-15T20:48:47.065 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.065 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.065 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:47.067943+0000 2024-09-15T20:48:47.066 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:47.067943+0000 2024-09-15T20:48:47.066 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.066 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.066 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.066 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:47.066 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:47.066 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 Processor -- start 2024-09-15T20:48:47.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- start start 2024-09-15T20:48:47.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient: init 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient(hunting): picked mon.noname-a con 0x7f4b180a0460 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient(hunting): picked mon.noname-b con 0x7f4b18137830 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient(hunting): picked mon.noname-c con 0x7f4b18137cc0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b18074310 con 0x7f4b180a0460 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b18072fa0 con 0x7f4b18137cc0 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b18073770 con 0x7f4b18137830 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient(hunting): authenticate will time out at 2187.953083s 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b16ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4b180a0460 0x7f4b180a0840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35090/0 (socket says 172.21.15.78:35090) 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b16ffd640 1 -- 172.21.15.78:0/682294811 learned_addr learned my addr 172.21.15.78:0/682294811 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b167fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f4b18137830 0x7f4b1813cbb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33830/0 (socket says 172.21.15.78:33830) 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3442772871 0 0) 0x7f4b18073770 con 0x7f4b18137830 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient(hunting): my global_id is 154100 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b00003530 con 0x7f4b18137830 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262569291 0 0) 0x7f4b18074310 con 0x7f4b180a0460 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient(hunting): my global_id is 144177 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b18073770 con 0x7f4b180a0460 2024-09-15T20:48:47.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3726011841 0 0) 0x7f4b00003530 con 0x7f4b18137830 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 >> v1:172.21.15.146:6789/0 conn(0x7f4b18137cc0 legacy=0x7f4b181402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 >> v1:172.21.15.78:6789/0 conn(0x7f4b180a0460 legacy=0x7f4b180a0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 monclient: found mon.noname-b 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b18144a30 con 0x7f4b18137830 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b080016a0 con 0x7f4b18137830 2024-09-15T20:48:47.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 5 monclient: authenticate success, global_id 154100 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: dump: 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.077 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b080018c0 con 0x7f4b18137830 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 1 -- 172.21.15.78:0/682294811 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b08001d50 con 0x7f4b18137830 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 4 set_mon_vals no callback set 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b15ffb640 10 monclient: dump: 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.078 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient: shutdown 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/682294811 >> v1:172.21.15.78:6790/0 conn(0x7f4b18137830 legacy=0x7f4b1813cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/682294811 shutdown_connections 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/682294811 >> 172.21.15.78:0/682294811 conn(0x7f4b180a6ee0 msgr2=0x7f4b180a3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/682294811 shutdown_connections 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/682294811 wait complete. 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient: monmap: 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 librados: starting msgr at 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 librados: starting objecter 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 Processor -- start 2024-09-15T20:48:47.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 -- start start 2024-09-15T20:48:47.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 librados: setting wanted keys 2024-09-15T20:48:47.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 1 librados: calling monclient init 2024-09-15T20:48:47.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.067+0000 7f4b1d6e4640 10 monclient: init 2024-09-15T20:48:47.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:48:47.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient(hunting): picked mon.noname-b con 0x7f4b180a0460 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient(hunting): picked mon.noname-a con 0x7f4b18137830 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient(hunting): picked mon.noname-c con 0x7f4b18137cc0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b181415d0 con 0x7f4b18137830 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b18074c30 con 0x7f4b180a0460 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b1813adc0 con 0x7f4b18137cc0 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient(hunting): authenticate will time out at 2187.956244s 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b167fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f4b18137830 0x7f4b18138f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35102/0 (socket says 172.21.15.78:35102) 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b167fc640 1 -- 172.21.15.78:0/832142395 learned_addr learned my addr 172.21.15.78:0/832142395 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 client.?.objecter ms_handle_connect 0x7f4b18137830 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 client.?.objecter ms_handle_connect 0x7f4b18137cc0 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1953437448 0 0) 0x7f4b181415d0 con 0x7f4b18137830 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): my global_id is 144180 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b04004ec0 con 0x7f4b18137830 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2985130508 0 0) 0x7f4b1813adc0 con 0x7f4b18137cc0 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): my global_id is 154103 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b181415d0 con 0x7f4b18137cc0 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 client.?.objecter ms_handle_connect 0x7f4b180a0460 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 107149893 0 0) 0x7f4b04004ec0 con 0x7f4b18137830 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b1813adc0 con 0x7f4b18137830 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b08001cd0 con 0x7f4b18137830 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): dump: 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3743499301 0 0) 0x7f4b181415d0 con 0x7f4b18137cc0 2024-09-15T20:48:47.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4b04004ec0 con 0x7f4b18137cc0 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b10001c00 con 0x7f4b18137cc0 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): dump: 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.085 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484785771 0 0) 0x7f4b0c001900 con 0x7f4b180a0460 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient(hunting): my global_id is 134269 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b181415d0 con 0x7f4b180a0460 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1611419963 0 0) 0x7f4b1813adc0 con 0x7f4b18137830 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 >> v1:172.21.15.78:6790/0 conn(0x7f4b18137cc0 legacy=0x7f4b181d4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 >> v1:172.21.15.146:6789/0 conn(0x7f4b180a0460 legacy=0x7f4b18138870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 monclient: found mon.a 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b1813b0a0 con 0x7f4b18137830 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 5 monclient: authenticate success, global_id 144180 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient: _renew_subs 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4b18139690 con 0x7f4b18137830 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 client.144180.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient: _renew_subs 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f4b18134cc0 con 0x7f4b18137830 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 1 librados: init done 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b080011c0 con 0x7f4b18137830 2024-09-15T20:48:47.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b08004bc0 con 0x7f4b18137830 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 monclient: dump: 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.088 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b157fa640 4 set_mon_vals no callback set 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f4b08004de0 con 0x7f4b18137830 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4af8005090 con 0x7f4b18137830 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(687..687 src has 379..687) ==== 5918+0+0 (unknown 1991624602 0 0) 0x7f4b0804bb60 con 0x7f4b18137830 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 client.144180.objecter ms_dispatch 0x7f4b18000ba0 osd_map(687..687 src has 379..687) 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 3 client.144180.objecter handle_osd_map got epochs [687,687] > 0 2024-09-15T20:48:47.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 3 client.144180.objecter handle_osd_map decoding full epoch 687 2024-09-15T20:48:47.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 20 client.144180.objecter dump_active .. 0 homeless 2024-09-15T20:48:47.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4b16ffd640 10 monclient: get_auth_request con 0x7f4b0403e210 auth_method 0 2024-09-15T20:48:47.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.071+0000 7f4aff7fe640 10 client.144180.objecter ms_handle_connect 0x7f4b0403e210 2024-09-15T20:48:47.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.075+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4b08016c90 con 0x7f4b18137830 2024-09-15T20:48:47.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.075+0000 7f4aff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.075+0000 7f4aff7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:47.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.319+0000 7f4b1d6e4640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:48:47.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.319+0000 7f4b1d6e4640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:48:47.322 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.319+0000 7f4b1d6e4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.319+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4af8005380 con 0x7f4b18137830 2024-09-15T20:48:47.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4aff7fe640 1 -- 172.21.15.78:0/832142395 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 68025131) 0x7f4b08016590 con 0x7f4b18137830 2024-09-15T20:48:47.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4aff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:48:47.323 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4aff7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:47.323 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:48:47.325 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 10 librados: watch_flush enter 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 10 librados: watch_flush exit 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 20 client.144180.objecter shutdown clearing up homeless session... 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 10 client.144180.objecter successfully canceled tick 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 >> v1:172.21.15.146:6812/435598642 conn(0x7f4b0403e210 legacy=0x7f4b040406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 10 monclient: shutdown 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 >> v1:172.21.15.78:6789/0 conn(0x7f4b18137830 legacy=0x7f4b18138f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 shutdown_connections 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 >> 172.21.15.78:0/832142395 conn(0x7f4b180a6ee0 msgr2=0x7f4b18142c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 shutdown_connections 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 1 -- 172.21.15.78:0/832142395 wait complete. 2024-09-15T20:48:47.326 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.323+0000 7f4b1d6e4640 1 librados: shutdown 2024-09-15T20:48:47.340 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:48:47.340 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:48:47.341 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:48:47.459 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient: get_monmap_and_config 2024-09-15T20:48:47.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.460 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient: monmap: 2024-09-15T20:48:47.460 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.460 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:47.461385+0000 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:47.461385+0000 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 Processor -- start 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- start start 2024-09-15T20:48:47.461 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient: init 2024-09-15T20:48:47.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient(hunting): picked mon.noname-a con 0x7ff2a4135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient(hunting): picked mon.noname-b con 0x7ff2a40a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient(hunting): picked mon.noname-c con 0x7ff2a413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a40745e0 con 0x7ff2a4135b60 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a406ecf0 con 0x7ff2a413c220 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a406efd0 con 0x7ff2a40a24e0 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient(hunting): authenticate will time out at 2188.345985s 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff2a4135b60 0x7ff2a4135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35106/0 (socket says 172.21.15.78:35106) 2024-09-15T20:48:47.472 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a37fe640 1 -- 172.21.15.78:0/3695548395 learned_addr learned my addr 172.21.15.78:0/3695548395 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550649408 0 0) 0x7ff2a40745e0 con 0x7ff2a4135b60 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient(hunting): my global_id is 144183 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff28c003530 con 0x7ff2a4135b60 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2685656481 0 0) 0x7ff2a406efd0 con 0x7ff2a40a24e0 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient(hunting): my global_id is 154106 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2a40745e0 con 0x7ff2a40a24e0 2024-09-15T20:48:47.473 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3343083773 0 0) 0x7ff28c003530 con 0x7ff2a4135b60 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 >> v1:172.21.15.78:6790/0 conn(0x7ff2a40a24e0 legacy=0x7ff2a413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 >> v1:172.21.15.146:6789/0 conn(0x7ff2a413c220 legacy=0x7ff2a413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 monclient: found mon.noname-a 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a4136650 con 0x7ff2a4135b60 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff2980016a0 con 0x7ff2a4135b60 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 5 monclient: authenticate success, global_id 144183 2024-09-15T20:48:47.474 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: dump: 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.475 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff298001840 con 0x7ff2a4135b60 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 1 -- 172.21.15.78:0/3695548395 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff298001c10 con 0x7ff2a4135b60 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2a27fc640 10 monclient: dump: 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.476 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 4 set_mon_vals no callback set 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 10 monclient: shutdown 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- 172.21.15.78:0/3695548395 >> v1:172.21.15.78:6789/0 conn(0x7ff2a4135b60 legacy=0x7ff2a4135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- 172.21.15.78:0/3695548395 shutdown_connections 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- 172.21.15.78:0/3695548395 >> 172.21.15.78:0/3695548395 conn(0x7ff2a40a63c0 msgr2=0x7ff2a40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- 172.21.15.78:0/3695548395 shutdown_connections 2024-09-15T20:48:47.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.459+0000 7ff2aa179640 1 -- 172.21.15.78:0/3695548395 wait complete. 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: monmap: 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 librados: starting msgr at 2024-09-15T20:48:47.478 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 librados: starting objecter 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 Processor -- start 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 -- start start 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 librados: setting wanted keys 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 librados: calling monclient init 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: init 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient(hunting): picked mon.noname-b con 0x7ff2a40a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient(hunting): picked mon.noname-c con 0x7ff2a4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient(hunting): picked mon.noname-a con 0x7ff2a413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a4138e70 con 0x7ff2a413c220 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a413b000 con 0x7ff2a40a24e0 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2a4139b10 con 0x7ff2a4135b60 2024-09-15T20:48:47.479 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient(hunting): authenticate will time out at 2188.349192s 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2a2ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff2a4135b60 0x7ff2a4138760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33854/0 (socket says 172.21.15.78:33854) 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2a2ffd640 1 -- 172.21.15.78:0/1388284491 learned_addr learned my addr 172.21.15.78:0/1388284491 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2a3fff640 1 --1- 172.21.15.78:0/1388284491 >> v1:172.21.15.78:6789/0 conn(0x7ff2a413c220 0x7ff2a41d2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35108/0 (socket says 172.21.15.78:35108) 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.?.objecter ms_handle_connect 0x7ff2a4135b60 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.?.objecter ms_handle_connect 0x7ff2a413c220 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3713155455 0 0) 0x7ff2a4139b10 con 0x7ff2a4135b60 2024-09-15T20:48:47.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): my global_id is 154109 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff290004ec0 con 0x7ff2a4135b60 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1441709603 0 0) 0x7ff2a4138e70 con 0x7ff2a413c220 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): my global_id is 144186 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2a4139b10 con 0x7ff2a413c220 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.?.objecter ms_handle_connect 0x7ff2a40a24e0 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 260054380 0 0) 0x7ff2a4139b10 con 0x7ff2a413c220 2024-09-15T20:48:47.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2a4138e70 con 0x7ff2a413c220 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff29c001c40 con 0x7ff2a413c220 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): dump: 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.482 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2485298910 0 0) 0x7ff2a413b000 con 0x7ff2a40a24e0 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): my global_id is 134275 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2a4139b10 con 0x7ff2a40a24e0 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 704434742 0 0) 0x7ff290004ec0 con 0x7ff2a4135b60 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2a413b000 con 0x7ff2a4135b60 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff2940019a0 con 0x7ff2a4135b60 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): dump: 2024-09-15T20:48:47.483 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1024131423 0 0) 0x7ff2a4138e70 con 0x7ff2a413c220 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 >> v1:172.21.15.78:6790/0 conn(0x7ff2a4135b60 legacy=0x7ff2a4138760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 >> v1:172.21.15.146:6789/0 conn(0x7ff2a40a24e0 legacy=0x7ff2a4138050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.484 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 monclient: found mon.a 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2a411cb10 con 0x7ff2a413c220 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 5 monclient: authenticate success, global_id 144186 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: _renew_subs 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.485 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff2a413b8b0 con 0x7ff2a413c220 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 client.144186.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: _renew_subs 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff2a413a8d0 con 0x7ff2a413c220 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 librados: init done 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff29c0039a0 con 0x7ff2a413c220 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff29c003d30 con 0x7ff2a413c220 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2a1ffb640 4 set_mon_vals no callback set 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: dump: 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.486 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff2840065d0 con 0x7ff284002870 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.144186.objecter ms_handle_connect 0x7ff284002870 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.144186.objecter resend_mon_ops 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff29c003f50 con 0x7ff2a413c220 2024-09-15T20:48:47.487 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(688..688 src has 379..688) ==== 5913+0+0 (unknown 961957971 0 0) 0x7ff29c04bcc0 con 0x7ff2a413c220 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 client.144186.objecter ms_dispatch 0x7ff2a4000ba0 osd_map(688..688 src has 379..688) 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 3 client.144186.objecter handle_osd_map got epochs [688,688] > 0 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 3 client.144186.objecter handle_osd_map decoding full epoch 688 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 20 client.144186.objecter dump_active .. 0 homeless 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3229772807 0 0) 0x7ff2840065d0 con 0x7ff284002870 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient(hunting): my global_id is 144189 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2a4138e70 con 0x7ff284002870 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff2a2ffd640 10 monclient: get_auth_request con 0x7ff290042670 auth_method 0 2024-09-15T20:48:47.488 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2795798762 0 0) 0x7ff2a4138e70 con 0x7ff284002870 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff2840065d0 con 0x7ff284002870 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff298001540 con 0x7ff284002870 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 10 monclient: dump: 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.489 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 443992732 0 0) 0x7ff2840065d0 con 0x7ff284002870 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.463+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff28400a170 con 0x7ff284002870 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.467+0000 7ff28bfff640 10 client.144186.objecter ms_handle_connect 0x7ff290042670 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.467+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff298001460 con 0x7ff284002870 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.467+0000 7ff28bfff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.467+0000 7ff28bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.467+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 >> v1:172.21.15.78:6789/0 conn(0x7ff284002870 legacy=0x7ff284004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.490 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.467+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 >> v1:172.21.15.78:6789/0 conn(0x7ff284002870 legacy=0x7ff284004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff284009e30 con 0x7ff2840090f0 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 client.144186.objecter ms_handle_connect 0x7ff2840090f0 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 client.144186.objecter resend_mon_ops 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1670118976 0 0) 0x7ff284009e30 con 0x7ff2840090f0 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient(hunting): my global_id is 144192 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2840065d0 con 0x7ff2840090f0 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 219448213 0 0) 0x7ff2840065d0 con 0x7ff2840090f0 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff284009e30 con 0x7ff2840090f0 2024-09-15T20:48:47.492 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff298004950 con 0x7ff2840090f0 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient: dump: 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.493 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2410660566 0 0) 0x7ff284009e30 con 0x7ff2840090f0 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff28400c400 con 0x7ff2840090f0 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4307 (unknown 0 0 1230810108) 0x7ff298004950 con 0x7ff2840090f0 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 >> v1:172.21.15.78:6789/0 conn(0x7ff2840090f0 legacy=0x7ff284009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff28bfff640 1 -- 172.21.15.78:0/1388284491 >> v1:172.21.15.78:6789/0 conn(0x7ff2840090f0 legacy=0x7ff284009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:48:47.494 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 70, 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 129366, 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:48:47.495 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.496 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.497 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:48:47.498 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.499 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.500 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:48:47.501 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.502 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.503 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.504 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.505 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:48:47.506 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:48:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 10 librados: watch_flush enter 2024-09-15T20:48:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 10 librados: watch_flush exit 2024-09-15T20:48:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 20 client.144186.objecter shutdown clearing up homeless session... 2024-09-15T20:48:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 10 client.144186.objecter successfully canceled tick 2024-09-15T20:48:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 >> v1:172.21.15.146:6812/435598642 conn(0x7ff290042670 legacy=0x7ff290044b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.508 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 10 monclient: shutdown 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 >> v1:172.21.15.78:6789/0 conn(0x7ff2a413c220 legacy=0x7ff2a41d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2a3fff640 1 -- 172.21.15.78:0/1388284491 reap_dead start 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2a3fff640 1 -- 172.21.15.78:0/1388284491 reap_dead start 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 shutdown_connections 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 >> 172.21.15.78:0/1388284491 conn(0x7ff2a40a63c0 msgr2=0x7ff2a413f490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 shutdown_connections 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 1 -- 172.21.15.78:0/1388284491 wait complete. 2024-09-15T20:48:47.509 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.471+0000 7ff2aa179640 1 librados: shutdown 2024-09-15T20:48:47.509 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:48:47.510 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:48:47.510 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient: get_monmap_and_config 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient: monmap: 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:47.629748+0000 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:47.629748+0000 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.627 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.628 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.628 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:47.628 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:47.628 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 1 Processor -- start 2024-09-15T20:48:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 1 -- start start 2024-09-15T20:48:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient: init 2024-09-15T20:48:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:48:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient(hunting): picked mon.noname-a con 0x7f1b480a0ff0 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient(hunting): picked mon.noname-c con 0x7f1b48073800 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient(hunting): picked mon.noname-b con 0x7f1b480a1ae0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b48072df0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b48074110 con 0x7f1b48073800 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b480743f0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient(hunting): authenticate will time out at 2188.514694s 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b477fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1b480a0ff0 0x7f1b480a13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35126/0 (socket says 172.21.15.78:35126) 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b477fe640 1 -- 172.21.15.78:0/340538848 learned_addr learned my addr 172.21.15.78:0/340538848 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1061951543 0 0) 0x7f1b480743f0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): my global_id is 154112 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b30003530 con 0x7f1b480a1ae0 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3445538183 0 0) 0x7f1b48072df0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): my global_id is 144195 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b480743f0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2405395474 0 0) 0x7f1b48074110 con 0x7f1b48073800 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient(hunting): my global_id is 134278 2024-09-15T20:48:47.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b48072df0 con 0x7f1b48073800 2024-09-15T20:48:47.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 18076080 0 0) 0x7f1b480743f0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 >> v1:172.21.15.78:6790/0 conn(0x7f1b480a1ae0 legacy=0x7f1b48140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 >> v1:172.21.15.146:6789/0 conn(0x7f1b48073800 legacy=0x7f1b4813ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 monclient: found mon.noname-a 2024-09-15T20:48:47.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b48074eb0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1b3c001530 con 0x7f1b480a0ff0 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 5 monclient: authenticate success, global_id 144195 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: dump: 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.640 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1b3c0016d0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 1 -- 172.21.15.78:0/340538848 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1b3c001b60 con 0x7f1b480a0ff0 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:47.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b4dff9640 4 set_mon_vals no callback set 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.627+0000 7f1b467fc640 10 monclient: dump: 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: shutdown 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- 172.21.15.78:0/340538848 >> v1:172.21.15.78:6789/0 conn(0x7f1b480a0ff0 legacy=0x7f1b480a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- 172.21.15.78:0/340538848 shutdown_connections 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- 172.21.15.78:0/340538848 >> 172.21.15.78:0/340538848 conn(0x7f1b480a63c0 msgr2=0x7f1b480a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- 172.21.15.78:0/340538848 shutdown_connections 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- 172.21.15.78:0/340538848 wait complete. 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: monmap: 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.643 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 librados: starting msgr at 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 librados: starting objecter 2024-09-15T20:48:47.644 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 Processor -- start 2024-09-15T20:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- start start 2024-09-15T20:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 librados: setting wanted keys 2024-09-15T20:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 librados: calling monclient init 2024-09-15T20:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: init 2024-09-15T20:48:47.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:48:47.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient(hunting): picked mon.noname-b con 0x7f1b48073800 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.648 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient(hunting): picked mon.noname-c con 0x7f1b480a0ff0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient(hunting): picked mon.noname-a con 0x7f1b480a1ae0 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b4813a7b0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b481373c0 con 0x7f1b48073800 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b4809d180 con 0x7f1b480a0ff0 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient(hunting): authenticate will time out at 2188.517703s 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b46ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1b480a0ff0 0x7f1b48136cb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33866/0 (socket says 172.21.15.78:33866) 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b46ffd640 1 -- 172.21.15.78:0/1605706541 learned_addr learned my addr 172.21.15.78:0/1605706541 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b47fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1b480a1ae0 0x7f1b481d5130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35140/0 (socket says 172.21.15.78:35140) 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 client.?.objecter ms_handle_connect 0x7f1b480a0ff0 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.649 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 client.?.objecter ms_handle_connect 0x7f1b480a1ae0 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2387017763 0 0) 0x7f1b4813a7b0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): my global_id is 144198 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b34004ec0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1737840711 0 0) 0x7f1b4809d180 con 0x7f1b480a0ff0 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): my global_id is 154115 2024-09-15T20:48:47.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b4813a7b0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 client.?.objecter ms_handle_connect 0x7f1b48073800 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3005658953 0 0) 0x7f1b34004ec0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b4809d180 con 0x7f1b480a1ae0 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2225139176 0 0) 0x7f1b4813a7b0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b34004ec0 con 0x7f1b480a0ff0 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1b400017a0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): dump: 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.651 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1b38001880 con 0x7f1b480a0ff0 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): dump: 2024-09-15T20:48:47.652 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2938829907 0 0) 0x7f1b481373c0 con 0x7f1b48073800 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): my global_id is 134281 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b4813a7b0 con 0x7f1b48073800 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3078647382 0 0) 0x7f1b4809d180 con 0x7f1b480a1ae0 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 >> v1:172.21.15.78:6790/0 conn(0x7f1b480a0ff0 legacy=0x7f1b48136cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 >> v1:172.21.15.146:6789/0 conn(0x7f1b48073800 legacy=0x7f1b481365a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 monclient: found mon.a 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b481395e0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 5 monclient: authenticate success, global_id 144198 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: _renew_subs 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4813a4d0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 client.144198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: _renew_subs 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f1b481393a0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 librados: init done 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1b40003980 con 0x7f1b480a1ae0 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1b40003d90 con 0x7f1b480a1ae0 2024-09-15T20:48:47.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient: dump: 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b45ffb640 4 set_mon_vals no callback set 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1b280065d0 con 0x7f1b28002870 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 client.144198.objecter ms_handle_connect 0x7f1b28002870 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 client.144198.objecter resend_mon_ops 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3731613888 0 0) 0x7f1b280065d0 con 0x7f1b28002870 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 10 monclient(hunting): my global_id is 154118 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b4809d180 con 0x7f1b28002870 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.631+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1b40003f30 con 0x7f1b480a1ae0 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(688..688 src has 379..688) ==== 5913+0+0 (unknown 961957971 0 0) 0x7f1b4004baa0 con 0x7f1b480a1ae0 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 10 client.144198.objecter ms_dispatch 0x7f1b48000ba0 osd_map(688..688 src has 379..688) 2024-09-15T20:48:47.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 3 client.144198.objecter handle_osd_map got epochs [688,688] > 0 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 3 client.144198.objecter handle_osd_map decoding full epoch 688 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 20 client.144198.objecter dump_active .. 0 homeless 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 687982471 0 0) 0x7f1b4809d180 con 0x7f1b28002870 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b280065d0 con 0x7f1b28002870 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1b3c001600 con 0x7f1b28002870 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 10 monclient: dump: 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.658 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b46ffd640 10 monclient: get_auth_request con 0x7f1b3403e270 auth_method 0 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3200470484 0 0) 0x7f1b280065d0 con 0x7f1b28002870 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1b2800a170 con 0x7f1b28002870 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 10 client.144198.objecter ms_handle_connect 0x7f1b3403e270 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f1b3c001860 con 0x7f1b28002870 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 >> v1:172.21.15.78:6790/0 conn(0x7f1b28002870 legacy=0x7f1b28004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.635+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 >> v1:172.21.15.78:6790/0 conn(0x7f1b28002870 legacy=0x7f1b28004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:47.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b4dff9640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:48:47.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f1b28009e80 con 0x7f1b28009140 2024-09-15T20:48:47.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 client.144198.objecter ms_handle_connect 0x7f1b28009140 2024-09-15T20:48:47.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 client.144198.objecter resend_mon_ops 2024-09-15T20:48:47.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393078205 0 0) 0x7f1b28009e80 con 0x7f1b28009140 2024-09-15T20:48:47.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient(hunting): my global_id is 154121 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1b280065d0 con 0x7f1b28009140 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2871104639 0 0) 0x7f1b280065d0 con 0x7f1b28009140 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1b28009e80 con 0x7f1b28009140 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1b3c004990 con 0x7f1b28009140 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient: dump: 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.662 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 684125934 0 0) 0x7f1b28009e80 con 0x7f1b28009140 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f1b2800c400 con 0x7f1b28009140 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3970 (unknown 0 0 122427866) 0x7f1b3c005e20 con 0x7f1b28009140 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 >> v1:172.21.15.78:6790/0 conn(0x7f1b28009140 legacy=0x7f1b280095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.639+0000 7f1b2ffff640 1 -- 172.21.15.78:0/1605706541 >> v1:172.21.15.78:6790/0 conn(0x7f1b28009140 legacy=0x7f1b280095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:47.664 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:48:47.664 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:48:47.664 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:48:47.664 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 70, 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 15623, 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:47.665 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.666 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.667 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.668 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.668 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:48:47.668 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:48:47.668 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:48:47.669 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.670 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:48:47.671 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.672 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.673 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:48:47.674 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.675 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.676 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:48:47.677 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 10 librados: watch_flush enter 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 10 librados: watch_flush exit 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 20 client.144198.objecter shutdown clearing up homeless session... 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 10 client.144198.objecter successfully canceled tick 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 >> v1:172.21.15.146:6812/435598642 conn(0x7f1b3403e270 legacy=0x7f1b34040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 10 monclient: shutdown 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 >> v1:172.21.15.78:6789/0 conn(0x7f1b480a1ae0 legacy=0x7f1b481d5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b47fff640 1 -- 172.21.15.78:0/1605706541 reap_dead start 2024-09-15T20:48:47.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b47fff640 1 -- 172.21.15.78:0/1605706541 reap_dead start 2024-09-15T20:48:47.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 shutdown_connections 2024-09-15T20:48:47.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 >> 172.21.15.78:0/1605706541 conn(0x7f1b480a63c0 msgr2=0x7f1b48138170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 shutdown_connections 2024-09-15T20:48:47.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 1 -- 172.21.15.78:0/1605706541 wait complete. 2024-09-15T20:48:47.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.643+0000 7f1b4dff9640 1 librados: shutdown 2024-09-15T20:48:47.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:48:47.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:48:47.680 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:48:47.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient: get_monmap_and_config 2024-09-15T20:48:47.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.787 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient: monmap: 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:47.790076+0000 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:47.790076+0000 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:47.788 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 1 Processor -- start 2024-09-15T20:48:47.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 1 -- start start 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient: init 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient(hunting): picked mon.noname-c con 0x7fdf70135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient(hunting): picked mon.noname-a con 0x7fdf700a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient(hunting): picked mon.noname-b con 0x7fdf7013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf700745e0 con 0x7fdf700a24e0 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7006ecf0 con 0x7fdf70135b60 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7006efd0 con 0x7fdf7013c220 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient(hunting): authenticate will time out at 2188.674554s 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdf700a24e0 0x7fdf7013ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35144/0 (socket says 172.21.15.78:35144) 2024-09-15T20:48:47.799 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6ffff640 1 -- 172.21.15.78:0/1742366819 learned_addr learned my addr 172.21.15.78:0/1742366819 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 731709171 0 0) 0x7fdf700745e0 con 0x7fdf700a24e0 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient(hunting): my global_id is 144201 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf58003530 con 0x7fdf700a24e0 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 357065047 0 0) 0x7fdf7006efd0 con 0x7fdf7013c220 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient(hunting): my global_id is 154124 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf700745e0 con 0x7fdf7013c220 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1427327410 0 0) 0x7fdf58003530 con 0x7fdf700a24e0 2024-09-15T20:48:47.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 >> v1:172.21.15.78:6790/0 conn(0x7fdf7013c220 legacy=0x7fdf7013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 >> v1:172.21.15.146:6789/0 conn(0x7fdf70135b60 legacy=0x7fdf70135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 monclient: found mon.noname-a 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf70136650 con 0x7fdf700a24e0 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdf600016a0 con 0x7fdf700a24e0 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: dump: 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.801 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 5 monclient: authenticate success, global_id 144201 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf60001840 con 0x7fdf700a24e0 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 1 -- 172.21.15.78:0/1742366819 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdf60001c10 con 0x7fdf700a24e0 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.802 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf6f7fe640 10 monclient: dump: 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 4 set_mon_vals no callback set 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 10 monclient: shutdown 2024-09-15T20:48:47.803 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 1 -- 172.21.15.78:0/1742366819 >> v1:172.21.15.78:6789/0 conn(0x7fdf700a24e0 legacy=0x7fdf7013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 1 -- 172.21.15.78:0/1742366819 shutdown_connections 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.787+0000 7fdf77181640 1 -- 172.21.15.78:0/1742366819 >> 172.21.15.78:0/1742366819 conn(0x7fdf700a63c0 msgr2=0x7fdf700a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- 172.21.15.78:0/1742366819 shutdown_connections 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- 172.21.15.78:0/1742366819 wait complete. 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: monmap: 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.804 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 librados: starting msgr at 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 librados: starting objecter 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 Processor -- start 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- start start 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 librados: setting wanted keys 2024-09-15T20:48:47.805 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 librados: calling monclient init 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: init 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient(hunting): picked mon.noname-a con 0x7fdf700a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient(hunting): picked mon.noname-c con 0x7fdf70135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient(hunting): picked mon.noname-b con 0x7fdf7013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7009d180 con 0x7fdf700a24e0 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf701393d0 con 0x7fdf7013c220 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf7009ef80 con 0x7fdf70135b60 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient(hunting): authenticate will time out at 2188.677422s 2024-09-15T20:48:47.806 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf74f1d640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdf700a24e0 0x7fdf701cce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35154/0 (socket says 172.21.15.78:35154) 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf74f1d640 1 -- 172.21.15.78:0/2831135732 learned_addr learned my addr 172.21.15.78:0/2831135732 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.?.objecter ms_handle_connect 0x7fdf700a24e0 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.?.objecter ms_handle_connect 0x7fdf70135b60 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 558366897 0 0) 0x7fdf7009ef80 con 0x7fdf70135b60 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.807 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): my global_id is 154127 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf5c004ec0 con 0x7fdf70135b60 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1787197992 0 0) 0x7fdf7006ecf0 con 0x7fdf700a24e0 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): my global_id is 144204 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf7009ef80 con 0x7fdf700a24e0 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1842393278 0 0) 0x7fdf5c004ec0 con 0x7fdf70135b60 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf7006ecf0 con 0x7fdf70135b60 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.?.objecter ms_handle_connect 0x7fdf7013c220 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdf60001560 con 0x7fdf70135b60 2024-09-15T20:48:47.808 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): dump: 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:47.809 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1412160326 0 0) 0x7fdf7009ef80 con 0x7fdf700a24e0 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf5c004ec0 con 0x7fdf700a24e0 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdf64001980 con 0x7fdf700a24e0 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): dump: 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.810 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4052242730 0 0) 0x7fdf7006ecf0 con 0x7fdf70135b60 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 >> v1:172.21.15.146:6789/0 conn(0x7fdf7013c220 legacy=0x7fdf701d55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 >> v1:172.21.15.78:6789/0 conn(0x7fdf700a24e0 legacy=0x7fdf701cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 monclient: found mon.c 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf70139c70 con 0x7fdf70135b60 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 5 monclient: authenticate success, global_id 154127 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: _renew_subs 2024-09-15T20:48:47.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdf70135590 con 0x7fdf70135b60 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 client.154127.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: _renew_subs 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdf7009e4d0 con 0x7fdf70135b60 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 librados: init done 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdf60004aa0 con 0x7fdf70135b60 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdf60004df0 con 0x7fdf70135b60 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6effd640 4 set_mon_vals no callback set 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient: dump: 2024-09-15T20:48:47.812 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fdf60001700 con 0x7fdf70135b60 2024-09-15T20:48:47.813 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdf500065d0 con 0x7fdf50002870 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(688..688 src has 379..688) ==== 5913+0+0 (unknown 961957971 0 0) 0x7fdf6004b8b0 con 0x7fdf70135b60 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.154127.objecter ms_dispatch 0x7fdf70000ba0 osd_map(688..688 src has 379..688) 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 3 client.154127.objecter handle_osd_map got epochs [688,688] > 0 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 3 client.154127.objecter handle_osd_map decoding full epoch 688 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 20 client.154127.objecter dump_active .. 0 homeless 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.154127.objecter ms_handle_connect 0x7fdf50002870 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.154127.objecter resend_mon_ops 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf7571e640 10 monclient: get_auth_request con 0x7fdf5c03e080 auth_method 0 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1157745045 0 0) 0x7fdf500065d0 con 0x7fdf50002870 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 monclient(hunting): my global_id is 134290 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf7006ecf0 con 0x7fdf50002870 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.791+0000 7fdf6d7fa640 10 client.154127.objecter ms_handle_connect 0x7fdf5c03e080 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3076305712 0 0) 0x7fdf7006ecf0 con 0x7fdf50002870 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf500065d0 con 0x7fdf50002870 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdf640037a0 con 0x7fdf50002870 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 10 monclient: dump: 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.815 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 672116262 0 0) 0x7fdf500065d0 con 0x7fdf50002870 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf5000a170 con 0x7fdf50002870 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fdf640057d0 con 0x7fdf50002870 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 >> v1:172.21.15.146:6789/0 conn(0x7fdf50002870 legacy=0x7fdf50004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.795+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 >> v1:172.21.15.146:6789/0 conn(0x7fdf50002870 legacy=0x7fdf50004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.799+0000 7fdf77181640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.799+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fdf50009e30 con 0x7fdf500090f0 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.799+0000 7fdf6d7fa640 10 client.154127.objecter ms_handle_connect 0x7fdf500090f0 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.799+0000 7fdf6d7fa640 10 client.154127.objecter resend_mon_ops 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2484159060 0 0) 0x7fdf50009e30 con 0x7fdf500090f0 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient(hunting): my global_id is 134293 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf500065d0 con 0x7fdf500090f0 2024-09-15T20:48:47.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1630340760 0 0) 0x7fdf500065d0 con 0x7fdf500090f0 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdf50009e30 con 0x7fdf500090f0 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdf64003d90 con 0x7fdf500090f0 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient: dump: 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.819 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3599293189 0 0) 0x7fdf50009e30 con 0x7fdf500090f0 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf5000c400 con 0x7fdf500090f0 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 398858009) 0x7fdf64005980 con 0x7fdf500090f0 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 >> v1:172.21.15.146:6789/0 conn(0x7fdf500090f0 legacy=0x7fdf50009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf6d7fa640 1 -- 172.21.15.78:0/2831135732 >> v1:172.21.15.146:6789/0 conn(0x7fdf500090f0 legacy=0x7fdf50009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:48:47.820 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 70, 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 237671, 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.821 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.822 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:48:47.823 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:48:47.824 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:47.825 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:48:47.826 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:48:47.827 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.828 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.829 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.830 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:47.831 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:48:47.832 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:47.832 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:47.832 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:47.832 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:48:47.832 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:48:47.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 10 librados: watch_flush enter 2024-09-15T20:48:47.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 10 librados: watch_flush exit 2024-09-15T20:48:47.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 20 client.154127.objecter shutdown clearing up homeless session... 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 10 client.154127.objecter successfully canceled tick 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 >> v1:172.21.15.146:6812/435598642 conn(0x7fdf5c03e080 legacy=0x7fdf5c040540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 10 monclient: shutdown 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 >> v1:172.21.15.78:6790/0 conn(0x7fdf70135b60 legacy=0x7fdf701d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf7571e640 1 -- 172.21.15.78:0/2831135732 reap_dead start 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf7571e640 1 -- 172.21.15.78:0/2831135732 reap_dead start 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 shutdown_connections 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 >> 172.21.15.78:0/2831135732 conn(0x7fdf700a63c0 msgr2=0x7fdf7013f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 shutdown_connections 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 1 -- 172.21.15.78:0/2831135732 wait complete. 2024-09-15T20:48:47.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.803+0000 7fdf77181640 1 librados: shutdown 2024-09-15T20:48:47.835 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:48:47.835 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:48:47.835 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:48:47.940 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient: get_monmap_and_config 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient: monmap: 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:47.943464+0000 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:47.943464+0000 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:47.941 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 1 Processor -- start 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 1 -- start start 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient: init 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient(hunting): picked mon.noname-a con 0x7fef14134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient(hunting): picked mon.noname-b con 0x7fef1409f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient(hunting): picked mon.noname-c con 0x7fef1406e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef140a0470 con 0x7fef14134bb0 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef1409feb0 con 0x7fef1406e960 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef1409e4a0 con 0x7fef1409f5a0 2024-09-15T20:48:47.949 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.939+0000 7fef198a4640 10 monclient(hunting): authenticate will time out at 2188.827944s 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef12ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fef14134bb0 0x7fef14134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35160/0 (socket says 172.21.15.78:35160) 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef12ffd640 1 -- 172.21.15.78:0/1877967175 learned_addr learned my addr 172.21.15.78:0/1877967175 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2833735124 0 0) 0x7fef1409e4a0 con 0x7fef1409f5a0 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient(hunting): my global_id is 154130 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef00003530 con 0x7fef1409f5a0 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3248051853 0 0) 0x7fef0c001120 con 0x7fef14134bb0 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.950 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient(hunting): my global_id is 144207 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef1409e4a0 con 0x7fef14134bb0 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4225928501 0 0) 0x7fef1409e4a0 con 0x7fef14134bb0 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 >> v1:172.21.15.78:6790/0 conn(0x7fef1409f5a0 legacy=0x7fef141399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 >> v1:172.21.15.146:6789/0 conn(0x7fef1406e960 legacy=0x7fef1413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 monclient: found mon.noname-a 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef140a0ff0 con 0x7fef14134bb0 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.951 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fef0c001850 con 0x7fef14134bb0 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 5 monclient: authenticate success, global_id 144207 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: dump: 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.952 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.953 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.953 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.953 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.953 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.953 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fef0c001a70 con 0x7fef14134bb0 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 1 -- 172.21.15.78:0/1877967175 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fef0c004a50 con 0x7fef14134bb0 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 4 set_mon_vals no callback set 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef11ffb640 10 monclient: dump: 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.954 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: shutdown 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- 172.21.15.78:0/1877967175 >> v1:172.21.15.78:6789/0 conn(0x7fef14134bb0 legacy=0x7fef14134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- 172.21.15.78:0/1877967175 shutdown_connections 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- 172.21.15.78:0/1877967175 >> 172.21.15.78:0/1877967175 conn(0x7fef140a63c0 msgr2=0x7fef140a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:47.955 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- 172.21.15.78:0/1877967175 shutdown_connections 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- 172.21.15.78:0/1877967175 wait complete. 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: build_initial_monmap 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: monmap: 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:47.956 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 librados: starting msgr at 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 librados: starting objecter 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 Processor -- start 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- start start 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 librados: setting wanted keys 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 librados: calling monclient init 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: init 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:48:47.957 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient(hunting): picked mon.noname-c con 0x7fef1406e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient(hunting): picked mon.noname-a con 0x7fef1409f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient(hunting): picked mon.noname-b con 0x7fef14134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef14120280 con 0x7fef1409f5a0 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef1409d180 con 0x7fef14134bb0 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fef1411df20 con 0x7fef1406e960 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient(hunting): authenticate will time out at 2188.830780s 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef12ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fef1406e960 0x7fef141328a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33926/0 (socket says 172.21.15.78:33926) 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef127fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fef1409f5a0 0x7fef141cd7c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35166/0 (socket says 172.21.15.78:35166) 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef12ffd640 1 -- 172.21.15.78:0/1630148056 learned_addr learned my addr 172.21.15.78:0/1630148056 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 client.?.objecter ms_handle_connect 0x7fef1409f5a0 2024-09-15T20:48:47.958 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 client.?.objecter ms_handle_connect 0x7fef1406e960 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1977361552 0 0) 0x7fef14120280 con 0x7fef1409f5a0 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): my global_id is 144210 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feef4004ec0 con 0x7fef1409f5a0 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2158314137 0 0) 0x7fef1411df20 con 0x7fef1406e960 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): my global_id is 154133 2024-09-15T20:48:47.959 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef14120280 con 0x7fef1406e960 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 client.?.objecter ms_handle_connect 0x7fef14134bb0 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3482644160 0 0) 0x7feef4004ec0 con 0x7fef1409f5a0 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fef1411df20 con 0x7fef1409f5a0 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feefc001420 con 0x7fef1409f5a0 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): dump: 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.960 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3353312077 0 0) 0x7fef14120280 con 0x7fef1406e960 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feef4004ec0 con 0x7fef1406e960 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fef0c001190 con 0x7fef1406e960 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): dump: 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.961 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3235499430 0 0) 0x7fef1409d180 con 0x7fef14134bb0 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient(hunting): my global_id is 134299 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fef14120280 con 0x7fef14134bb0 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4134448893 0 0) 0x7fef1411df20 con 0x7fef1409f5a0 2024-09-15T20:48:47.962 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 >> v1:172.21.15.78:6790/0 conn(0x7fef1406e960 legacy=0x7fef141328a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 >> v1:172.21.15.146:6789/0 conn(0x7fef14134bb0 legacy=0x7fef141d0ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 monclient: found mon.a 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef1411ec00 con 0x7fef1409f5a0 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient: _finish_auth 0 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 5 monclient: authenticate success, global_id 144210 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: _renew_subs 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fef1411ffa0 con 0x7fef1409f5a0 2024-09-15T20:48:47.963 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 client.144210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: _renew_subs 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fef14133180 con 0x7fef1409f5a0 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef198a4640 1 librados: init done 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7feefc003600 con 0x7fef1409f5a0 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7feefc003a10 con 0x7fef1409f5a0 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef117fa640 4 set_mon_vals no callback set 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.943+0000 7fef0bfff640 10 monclient: dump: 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.964 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef198a4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef198a4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef198a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feef8005090 con 0x7fef1409f5a0 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7feefc003bb0 con 0x7fef1409f5a0 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(688..688 src has 379..688) ==== 5913+0+0 (unknown 961957971 0 0) 0x7feefc04b960 con 0x7fef1409f5a0 2024-09-15T20:48:47.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef0bfff640 10 client.144210.objecter ms_dispatch 0x7fef14000ba0 osd_map(688..688 src has 379..688) 2024-09-15T20:48:47.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef0bfff640 3 client.144210.objecter handle_osd_map got epochs [688,688] > 0 2024-09-15T20:48:47.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef0bfff640 3 client.144210.objecter handle_osd_map decoding full epoch 688 2024-09-15T20:48:47.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef0bfff640 20 client.144210.objecter dump_active .. 0 homeless 2024-09-15T20:48:47.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef12ffd640 10 monclient: get_auth_request con 0x7feef403e210 auth_method 0 2024-09-15T20:48:47.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.947+0000 7fef0bfff640 10 client.144210.objecter ms_handle_connect 0x7feef403e210 2024-09-15T20:48:47.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.951+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7feefc0169d0 con 0x7fef1409f5a0 2024-09-15T20:48:47.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.951+0000 7fef0bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:47.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:47.951+0000 7fef0bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:48.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.171+0000 7fef198a4640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:48:48.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.171+0000 7fef198a4640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:48:48.173 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.171+0000 7fef198a4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:48.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.171+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7feef8005c50 con 0x7fef1409f5a0 2024-09-15T20:48:48.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.175+0000 7fef0bfff640 1 -- 172.21.15.78:0/1630148056 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7feefc01adb0 con 0x7fef1409f5a0 2024-09-15T20:48:48.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.175+0000 7fef0bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:48:48.174 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.175+0000 7fef0bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 10 librados: watch_flush enter 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 10 librados: watch_flush exit 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 20 client.144210.objecter shutdown clearing up homeless session... 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 10 client.144210.objecter successfully canceled tick 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 >> v1:172.21.15.146:6812/435598642 conn(0x7feef403e210 legacy=0x7feef40406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 10 monclient: shutdown 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 >> v1:172.21.15.78:6789/0 conn(0x7fef1409f5a0 legacy=0x7fef141cd7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:48.179 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 shutdown_connections 2024-09-15T20:48:48.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 >> 172.21.15.78:0/1630148056 conn(0x7fef140a63c0 msgr2=0x7fef1409d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:48.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 shutdown_connections 2024-09-15T20:48:48.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 1 -- 172.21.15.78:0/1630148056 wait complete. 2024-09-15T20:48:48.180 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fef198a4640 1 librados: shutdown 2024-09-15T20:48:48.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=12,logm=50} crc {auth=959947243,config=2586702516,health=1157775369,kv=4191261044,logm=4236472918}) 2024-09-15T20:48:48.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=23,config=2,health=12,kv=12,logm=51} crc {auth=2611544430,config=2586702516,health=1157775369,kv=4051999118,logm=4031920568}) 2024-09-15T20:48:48.181 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.181 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=12,kv=12,logm=50} crc {auth=959947243,config=2586702516,health=1157775369,kv=4191261044,logm=4236472918}) 2024-09-15T20:48:48.181 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.179+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=11,kv=12,logm=50} crc {auth=1316858224,config=2586702516,health=4181706558,kv=4051999118,logm=2506683256}) 2024-09-15T20:48:48.182 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.182 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220760939}) 2024-09-15T20:48:48.182 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2892377897}) 2024-09-15T20:48:48.183 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.183 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1220760939}) 2024-09-15T20:48:48.183 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3918960486}) 2024-09-15T20:48:48.184 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424040671}) 2024-09-15T20:48:48.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2241824041}) 2024-09-15T20:48:48.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=424040671}) 2024-09-15T20:48:48.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.183+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2499329122}) 2024-09-15T20:48:48.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.187+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.187+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439222156}) 2024-09-15T20:48:48.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.187+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2084527102}) 2024-09-15T20:48:48.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.187+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.187+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=439222156}) 2024-09-15T20:48:48.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.187+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2891373020}) 2024-09-15T20:48:48.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256688903}) 2024-09-15T20:48:48.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1394652202}) 2024-09-15T20:48:48.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3256688903}) 2024-09-15T20:48:48.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3875449443}) 2024-09-15T20:48:48.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598401181}) 2024-09-15T20:48:48.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4222518417}) 2024-09-15T20:48:48.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.193 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3598401181}) 2024-09-15T20:48:48.193 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.191+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2252291334}) 2024-09-15T20:48:48.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=55587581}) 2024-09-15T20:48:48.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=254841190}) 2024-09-15T20:48:48.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=55587581}) 2024-09-15T20:48:48.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2751458738}) 2024-09-15T20:48:48.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412366238}) 2024-09-15T20:48:48.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1614466050}) 2024-09-15T20:48:48.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412366238}) 2024-09-15T20:48:48.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=130306255}) 2024-09-15T20:48:48.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897414407}) 2024-09-15T20:48:48.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3692596751}) 2024-09-15T20:48:48.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.195+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3897414407}) 2024-09-15T20:48:48.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=102260736}) 2024-09-15T20:48:48.200 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:48:48.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498662668}) 2024-09-15T20:48:48.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1868645077}) 2024-09-15T20:48:48.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1498662668}) 2024-09-15T20:48:48.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=334735017}) 2024-09-15T20:48:48.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209174678}) 2024-09-15T20:48:48.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.199+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1341220060}) 2024-09-15T20:48:48.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=209174678}) 2024-09-15T20:48:48.203 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3698599737}) 2024-09-15T20:48:48.205 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.205 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187952137}) 2024-09-15T20:48:48.205 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=71583329}) 2024-09-15T20:48:48.205 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.205 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2187952137}) 2024-09-15T20:48:48.205 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.203+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1381240393}) 2024-09-15T20:48:48.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4108455265}) 2024-09-15T20:48:48.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2357658379}) 2024-09-15T20:48:48.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4108455265}) 2024-09-15T20:48:48.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2671185578}) 2024-09-15T20:48:48.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901744105}) 2024-09-15T20:48:48.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2469227770}) 2024-09-15T20:48:48.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2901744105}) 2024-09-15T20:48:48.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.207+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4146638087}) 2024-09-15T20:48:48.210 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884651803}) 2024-09-15T20:48:48.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2228470085}) 2024-09-15T20:48:48.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1884651803}) 2024-09-15T20:48:48.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2940646853}) 2024-09-15T20:48:48.212 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276953215}) 2024-09-15T20:48:48.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1636485630}) 2024-09-15T20:48:48.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276953215}) 2024-09-15T20:48:48.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.211+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4186308584}) 2024-09-15T20:48:48.214 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.214 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1162304422}) 2024-09-15T20:48:48.214 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1714368006}) 2024-09-15T20:48:48.214 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.214 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1162304422}) 2024-09-15T20:48:48.214 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1306645074}) 2024-09-15T20:48:48.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1608647621}) 2024-09-15T20:48:48.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3812949466}) 2024-09-15T20:48:48.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1608647621}) 2024-09-15T20:48:48.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3366150514}) 2024-09-15T20:48:48.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.215+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911417744}) 2024-09-15T20:48:48.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3171862289}) 2024-09-15T20:48:48.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2911417744}) 2024-09-15T20:48:48.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3439658919}) 2024-09-15T20:48:48.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574881726}) 2024-09-15T20:48:48.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1677567018}) 2024-09-15T20:48:48.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3574881726}) 2024-09-15T20:48:48.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4179060711}) 2024-09-15T20:48:48.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069168484}) 2024-09-15T20:48:48.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4153934307}) 2024-09-15T20:48:48.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1069168484}) 2024-09-15T20:48:48.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=815919175}) 2024-09-15T20:48:48.222 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.222 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281208277}) 2024-09-15T20:48:48.222 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1540167302}) 2024-09-15T20:48:48.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4281208277}) 2024-09-15T20:48:48.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.219+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3537918574}) 2024-09-15T20:48:48.224 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.224 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101417774}) 2024-09-15T20:48:48.224 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1953168554}) 2024-09-15T20:48:48.224 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.224 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=101417774}) 2024-09-15T20:48:48.224 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2811425497}) 2024-09-15T20:48:48.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063298011}) 2024-09-15T20:48:48.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=740813154}) 2024-09-15T20:48:48.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2063298011}) 2024-09-15T20:48:48.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.223+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3618868555}) 2024-09-15T20:48:48.235 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.235 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804531876}) 2024-09-15T20:48:48.235 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1687444917}) 2024-09-15T20:48:48.235 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.235 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2804531876}) 2024-09-15T20:48:48.235 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=474710550}) 2024-09-15T20:48:48.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1566955799}) 2024-09-15T20:48:48.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=663348210}) 2024-09-15T20:48:48.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1566955799}) 2024-09-15T20:48:48.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1859554463}) 2024-09-15T20:48:48.237 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.237 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441049546}) 2024-09-15T20:48:48.237 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1037965004}) 2024-09-15T20:48:48.237 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2441049546}) 2024-09-15T20:48:48.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.235+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=556800331}) 2024-09-15T20:48:48.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881723191}) 2024-09-15T20:48:48.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2039328794}) 2024-09-15T20:48:48.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1881723191}) 2024-09-15T20:48:48.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1693033759}) 2024-09-15T20:48:48.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.239 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30843377}) 2024-09-15T20:48:48.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1063810815}) 2024-09-15T20:48:48.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=30843377}) 2024-09-15T20:48:48.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3386010796}) 2024-09-15T20:48:48.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.241 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733296695}) 2024-09-15T20:48:48.241 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=341320716}) 2024-09-15T20:48:48.241 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.241 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1733296695}) 2024-09-15T20:48:48.241 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2834826308}) 2024-09-15T20:48:48.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107420193}) 2024-09-15T20:48:48.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1132291624}) 2024-09-15T20:48:48.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4107420193}) 2024-09-15T20:48:48.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.239+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2534161481}) 2024-09-15T20:48:48.243 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.243 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086988299}) 2024-09-15T20:48:48.243 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1376112799}) 2024-09-15T20:48:48.243 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.243 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4086988299}) 2024-09-15T20:48:48.243 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2415453486}) 2024-09-15T20:48:48.244 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.244 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=233387019}) 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=684530226}) 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=233387019}) 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3946325479}) 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094390910}) 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=11379318}) 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1094390910}) 2024-09-15T20:48:48.245 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2765474782}) 2024-09-15T20:48:48.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359217329}) 2024-09-15T20:48:48.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=732065148}) 2024-09-15T20:48:48.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1359217329}) 2024-09-15T20:48:48.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.243+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=230538439}) 2024-09-15T20:48:48.247 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.247 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386344293}) 2024-09-15T20:48:48.247 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2584975517}) 2024-09-15T20:48:48.247 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.247 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2386344293}) 2024-09-15T20:48:48.247 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2418709864}) 2024-09-15T20:48:48.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945226294}) 2024-09-15T20:48:48.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=90267440}) 2024-09-15T20:48:48.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2945226294}) 2024-09-15T20:48:48.249 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1060347568}) 2024-09-15T20:48:48.249 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.249 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065431075}) 2024-09-15T20:48:48.249 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4210168882}) 2024-09-15T20:48:48.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2065431075}) 2024-09-15T20:48:48.250 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.247+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2961345454}) 2024-09-15T20:48:48.251 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.251 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161352417}) 2024-09-15T20:48:48.251 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2941245356}) 2024-09-15T20:48:48.251 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.251 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=161352417}) 2024-09-15T20:48:48.251 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3884253572}) 2024-09-15T20:48:48.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479710492}) 2024-09-15T20:48:48.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=470677959}) 2024-09-15T20:48:48.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=479710492}) 2024-09-15T20:48:48.252 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2258853398}) 2024-09-15T20:48:48.253 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.253 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310216236}) 2024-09-15T20:48:48.254 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=741719415}) 2024-09-15T20:48:48.254 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.254 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3310216236}) 2024-09-15T20:48:48.254 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.251+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3484576690}) 2024-09-15T20:48:48.254 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.254 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704427958}) 2024-09-15T20:48:48.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2170842894}) 2024-09-15T20:48:48.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2704427958}) 2024-09-15T20:48:48.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3403060731}) 2024-09-15T20:48:48.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.255 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582585285}) 2024-09-15T20:48:48.256 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1769525218}) 2024-09-15T20:48:48.256 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.256 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1582585285}) 2024-09-15T20:48:48.256 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3238516735}) 2024-09-15T20:48:48.256 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.256 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107430941}) 2024-09-15T20:48:48.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1432127908}) 2024-09-15T20:48:48.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3107430941}) 2024-09-15T20:48:48.257 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=492294343}) 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380683584}) 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1560423849}) 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3380683584}) 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3746268429}) 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=847221305}) 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3032295536}) 2024-09-15T20:48:48.260 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.261 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=847221305}) 2024-09-15T20:48:48.261 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.255+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2188012030}) 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094213129}) 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3409049125}) 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3094213129}) 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2069190491}) 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426286717}) 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782543928}) 2024-09-15T20:48:48.262 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=426286717}) 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4141737791}) 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541795960}) 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3925941626}) 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3541795960}) 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=346125033}) 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676440353}) 2024-09-15T20:48:48.263 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2632726320}) 2024-09-15T20:48:48.264 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.264 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2676440353}) 2024-09-15T20:48:48.264 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.259+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3515600798}) 2024-09-15T20:48:48.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333559969}) 2024-09-15T20:48:48.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1236077290}) 2024-09-15T20:48:48.265 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=333559969}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1571835968}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572560429}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2564866847}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1572560429}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=484017794}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051306188}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2720340248}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3051306188}) 2024-09-15T20:48:48.266 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2904941749}) 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801223454}) 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2660784894}) 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2801223454}) 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3808176446}) 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234325852}) 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3286387562}) 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3234325852}) 2024-09-15T20:48:48.267 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.263+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1986672432}) 2024-09-15T20:48:48.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643843667}) 2024-09-15T20:48:48.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=897700477}) 2024-09-15T20:48:48.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3643843667}) 2024-09-15T20:48:48.269 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2600434766}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132505696}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2691118438}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4132505696}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=75327154}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223333735}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4164514105}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1223333735}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3060567085}) 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.270 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683768721}) 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=97252982}) 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=683768721}) 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2771888055}) 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114534080}) 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4032724754}) 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2114534080}) 2024-09-15T20:48:48.271 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.267+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2777611855}) 2024-09-15T20:48:48.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=622858919}) 2024-09-15T20:48:48.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2639644422}) 2024-09-15T20:48:48.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=622858919}) 2024-09-15T20:48:48.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3715822858}) 2024-09-15T20:48:48.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.273 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678670409}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1779523858}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3678670409}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4233207320}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381024952}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=86466644}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1381024952}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2809980783}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449624195}) 2024-09-15T20:48:48.274 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1598263434}) 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1449624195}) 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4202137478}) 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402586797}) 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1954288039}) 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3402586797}) 2024-09-15T20:48:48.275 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.271+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967215604}) 2024-09-15T20:48:48.276 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.276 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805112629}) 2024-09-15T20:48:48.276 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1390325581}) 2024-09-15T20:48:48.276 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1805112629}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2079442886}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246730507}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2199634265}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4246730507}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=820495854}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067060898}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2338216536}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2067060898}) 2024-09-15T20:48:48.277 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3647747376}) 2024-09-15T20:48:48.278 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.278 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711253463}) 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=273060042}) 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1711253463}) 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.275+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1892938351}) 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880971787}) 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=550958001}) 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2880971787}) 2024-09-15T20:48:48.279 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=662219126}) 2024-09-15T20:48:48.280 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.280 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=70,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7} crc {logm=3577836398,mdsmap=503983409,mgr=3591229687,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=342064145,mon_config_key=3991244134}) 2024-09-15T20:48:48.280 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=10,mon_config_key=8} crc {logm=3577908663,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=2675636619,mon_config_key=3731732939}) 2024-09-15T20:48:48.280 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.280 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=70,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7} crc {logm=3577836398,mdsmap=503983409,mgr=3591229687,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=342064145,mon_config_key=3991244134}) 2024-09-15T20:48:48.280 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=69,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=7} crc {logm=266500845,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=342064145,mon_config_key=3991244134}) 2024-09-15T20:48:48.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=10,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=41} crc {mon_config_key=2709962611,monmap=3202364208,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=2543971073,osd_snap=879496410,osdmap=3444581857}) 2024-09-15T20:48:48.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {mon_config_key=9,monmap=3,nvmeofgw=3,osd_metadata=5,osd_snap=40,osdmap=40} crc {mon_config_key=3342132884,monmap=3785329752,nvmeofgw=598337686,osd_metadata=3161001510,osd_snap=2479711666,osdmap=4238562500}) 2024-09-15T20:48:48.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.281 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {mon_config_key=10,monmap=2,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=37,osdmap=41} crc {mon_config_key=2709962611,monmap=3202364208,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=2543971073,osd_snap=879496410,osdmap=3444581857}) 2024-09-15T20:48:48.282 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.279+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {mon_config_key=10,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=37,osdmap=41} crc {mon_config_key=2709962611,monmap=3785329752,nvmeofgw=598337686,osd_metadata=159488793,osd_pg_creating=2543971073,osd_snap=2307024607,osdmap=469744992}) 2024-09-15T20:48:48.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2060274649}) 2024-09-15T20:48:48.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=778642878}) 2024-09-15T20:48:48.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=2060274649}) 2024-09-15T20:48:48.283 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=4250714163}) 2024-09-15T20:48:48.284 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.284 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=974611087}) 2024-09-15T20:48:48.284 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=113621692}) 2024-09-15T20:48:48.284 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.285 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=974611087}) 2024-09-15T20:48:48.285 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.283+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=70112834}) 2024-09-15T20:48:48.286 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.286 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3379511685}) 2024-09-15T20:48:48.286 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3613782489}) 2024-09-15T20:48:48.286 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.286 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=3379511685}) 2024-09-15T20:48:48.286 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3040096814}) 2024-09-15T20:48:48.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=830881526}) 2024-09-15T20:48:48.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=3090896524}) 2024-09-15T20:48:48.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.289 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=830881526}) 2024-09-15T20:48:48.290 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1802784289}) 2024-09-15T20:48:48.290 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.290 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=2588213020}) 2024-09-15T20:48:48.290 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=16} crc {osdmap=535264569}) 2024-09-15T20:48:48.290 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:48:48.290 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=17} crc {osdmap=2588213020}) 2024-09-15T20:48:48.291 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:48:48.287+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=15} crc {osdmap=62664167}) 2024-09-15T20:48:49.201 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:48:49.201 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.339+0000 7fc352d77640 10 monclient: get_monmap_and_config 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.339+0000 7fc352d77640 10 monclient: build_initial_monmap 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.339+0000 7fc352d77640 10 monclient: monmap: 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:49.343555+0000 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:49.343555+0000 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:49.341 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:49.342 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:49.342 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.339+0000 7fc352d77640 1 Processor -- start 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.339+0000 7fc352d77640 1 -- start start 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.339+0000 7fc352d77640 10 monclient: init 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): picked mon.noname-b con 0x7fc34c137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): picked mon.noname-c con 0x7fc34c0a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): picked mon.noname-a con 0x7fc34c13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34c0a6420 con 0x7fc34c13e5f0 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34c0a6700 con 0x7fc34c0a9610 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34c0a2110 con 0x7fc34c137ce0 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): authenticate will time out at 2190.228162s 2024-09-15T20:48:49.351 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc350b13640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc34c137ce0 0x7fc34c1380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33932/0 (socket says 172.21.15.78:33932) 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc350b13640 1 -- 172.21.15.78:0/637031155 learned_addr learned my addr 172.21.15.78:0/637031155 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070169239 0 0) 0x7fc34c0a2110 con 0x7fc34c137ce0 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient(hunting): my global_id is 154136 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc338003530 con 0x7fc34c137ce0 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2742164769 0 0) 0x7fc34c0a6420 con 0x7fc34c13e5f0 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.352 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient(hunting): my global_id is 144213 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc34c0a2110 con 0x7fc34c13e5f0 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2970533588 0 0) 0x7fc338003530 con 0x7fc34c137ce0 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 >> v1:172.21.15.146:6789/0 conn(0x7fc34c0a9610 legacy=0x7fc34c13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 >> v1:172.21.15.78:6789/0 conn(0x7fc34c13e5f0 legacy=0x7fc34c1409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 monclient: found mon.noname-b 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:48:49.353 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34c0a4bc0 con 0x7fc34c137ce0 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: _finish_auth 0 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc344001a20 con 0x7fc34c137ce0 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 5 monclient: authenticate success, global_id 154136 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: dump: 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.354 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc344001c40 con 0x7fc34c137ce0 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 1 -- 172.21.15.78:0/637031155 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc344002010 con 0x7fc34c137ce0 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 4 set_mon_vals no callback set 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc34b7fe640 10 monclient: dump: 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: shutdown 2024-09-15T20:48:49.356 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- 172.21.15.78:0/637031155 >> v1:172.21.15.78:6790/0 conn(0x7fc34c137ce0 legacy=0x7fc34c1380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- 172.21.15.78:0/637031155 shutdown_connections 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- 172.21.15.78:0/637031155 >> 172.21.15.78:0/637031155 conn(0x7fc34c0aa6c0 msgr2=0x7fc34c0aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- 172.21.15.78:0/637031155 shutdown_connections 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- 172.21.15.78:0/637031155 wait complete. 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: build_initial_monmap 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: monmap: 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.357 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:49.358 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 librados: starting msgr at 2024-09-15T20:48:49.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 librados: starting objecter 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 Processor -- start 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- start start 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 librados: setting wanted keys 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 librados: calling monclient init 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: init 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): picked mon.noname-a con 0x7fc34c0a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): picked mon.noname-b con 0x7fc34c137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): picked mon.noname-c con 0x7fc34c13e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34c0a14b0 con 0x7fc34c0a9610 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34c1392c0 con 0x7fc34c137ce0 2024-09-15T20:48:49.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc34c09e500 con 0x7fc34c13e5f0 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc352d77640 10 monclient(hunting): authenticate will time out at 2190.231481s 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc350b13640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc34c0a9610 0x7fc34c09cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35186/0 (socket says 172.21.15.78:35186) 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc350b13640 1 -- 172.21.15.78:0/3362950655 learned_addr learned my addr 172.21.15.78:0/3362950655 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 client.?.objecter ms_handle_connect 0x7fc34c0a9610 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 client.?.objecter ms_handle_connect 0x7fc34c13e5f0 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 985389222 0 0) 0x7fc34c0a14b0 con 0x7fc34c0a9610 2024-09-15T20:48:49.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 monclient(hunting): my global_id is 144216 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc32c004ec0 con 0x7fc34c0a9610 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1356289879 0 0) 0x7fc34c09e500 con 0x7fc34c13e5f0 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 monclient(hunting): my global_id is 154139 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc34c0a14b0 con 0x7fc34c13e5f0 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 client.?.objecter ms_handle_connect 0x7fc34c137ce0 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.343+0000 7fc3497fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1348055377 0 0) 0x7fc32c004ec0 con 0x7fc34c0a9610 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc34c09e500 con 0x7fc34c0a9610 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2324363795 0 0) 0x7fc34c1392c0 con 0x7fc34c137ce0 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): my global_id is 134305 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc32c004ec0 con 0x7fc34c137ce0 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc344001f90 con 0x7fc34c0a9610 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): dump: 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.362 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3860266749 0 0) 0x7fc34c0a14b0 con 0x7fc34c13e5f0 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc34c1392c0 con 0x7fc34c13e5f0 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc33c0018c0 con 0x7fc34c13e5f0 2024-09-15T20:48:49.363 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient(hunting): dump: 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1308202259 0 0) 0x7fc34c09e500 con 0x7fc34c0a9610 2024-09-15T20:48:49.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 >> v1:172.21.15.78:6790/0 conn(0x7fc34c13e5f0 legacy=0x7fc34c09da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 >> v1:172.21.15.146:6789/0 conn(0x7fc34c137ce0 legacy=0x7fc34c09d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient: _finish_hunting 0 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 monclient: found mon.a 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34c0a0860 con 0x7fc34c0a9610 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient: _finish_auth 0 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient: _check_auth_tickets 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 5 monclient: authenticate success, global_id 144216 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 10 monclient: _renew_subs 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:49.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc34c0a0580 con 0x7fc34c0a9610 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 10 client.144216.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 10 monclient: _renew_subs 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc34c1d5ff0 con 0x7fc34c0a9610 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 1 librados: init done 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc344004f80 con 0x7fc34c0a9610 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc3440053d0 con 0x7fc34c0a9610 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc34affd640 4 set_mon_vals no callback set 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 monclient: dump: 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.366 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc330005090 con 0x7fc34c0a9610 2024-09-15T20:48:49.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fc344005570 con 0x7fc34c0a9610 2024-09-15T20:48:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7fc34404bd60 con 0x7fc34c0a9610 2024-09-15T20:48:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 client.144216.objecter ms_dispatch 0x7fc34c000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:48:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 3 client.144216.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:48:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 3 client.144216.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:48:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 20 client.144216.objecter dump_active .. 0 homeless 2024-09-15T20:48:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc34bfff640 10 monclient: get_auth_request con 0x7fc32c03e180 auth_method 0 2024-09-15T20:48:49.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.347+0000 7fc3497fa640 10 client.144216.objecter ms_handle_connect 0x7fc32c03e180 2024-09-15T20:48:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.351+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc344016fb0 con 0x7fc34c0a9610 2024-09-15T20:48:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.351+0000 7fc3497fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:49.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.351+0000 7fc3497fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:49.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.591+0000 7fc352d77640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:48:49.591 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.591+0000 7fc352d77640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:48:49.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.591+0000 7fc352d77640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:49.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.591+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc330005380 con 0x7fc34c0a9610 2024-09-15T20:48:49.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.591+0000 7fc3497fa640 1 -- 172.21.15.78:0/3362950655 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 4221954526) 0x7fc3440168b0 con 0x7fc34c0a9610 2024-09-15T20:48:49.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.591+0000 7fc3497fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:48:49.592 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.591+0000 7fc3497fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:49.593 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 10 librados: watch_flush enter 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 10 librados: watch_flush exit 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 20 client.144216.objecter shutdown clearing up homeless session... 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 10 client.144216.objecter successfully canceled tick 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 >> v1:172.21.15.146:6812/435598642 conn(0x7fc32c03e180 legacy=0x7fc32c040640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 10 monclient: shutdown 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 >> v1:172.21.15.78:6789/0 conn(0x7fc34c0a9610 legacy=0x7fc34c09cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.595 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 shutdown_connections 2024-09-15T20:48:49.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 >> 172.21.15.78:0/3362950655 conn(0x7fc34c0aa6c0 msgr2=0x7fc34c0a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:49.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 shutdown_connections 2024-09-15T20:48:49.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 1 -- 172.21.15.78:0/3362950655 wait complete. 2024-09-15T20:48:49.596 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.595+0000 7fc352d77640 1 librados: shutdown 2024-09-15T20:48:49.611 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:48:49.611 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-09-15T20:48:49.611 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:48:49.612 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:48:49.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: get_monmap_and_config 2024-09-15T20:48:49.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: build_initial_monmap 2024-09-15T20:48:49.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: monmap: 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:49.737068+0000 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:49.737068+0000 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:49.735 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 Processor -- start 2024-09-15T20:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- start start 2024-09-15T20:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: init 2024-09-15T20:48:49.746 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient(hunting): picked mon.noname-b con 0x7f5234137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient(hunting): picked mon.noname-a con 0x7f52340a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient(hunting): picked mon.noname-c con 0x7f523413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52340a6420 con 0x7f52340a9610 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52340a6700 con 0x7f523413e5f0 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52340a2110 con 0x7f5234137ce0 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient(hunting): authenticate will time out at 2190.621668s 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52399d9640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f5234137ce0 0x7f52341380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33944/0 (socket says 172.21.15.78:33944) 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52399d9640 1 -- 172.21.15.78:0/4220264232 learned_addr learned my addr 172.21.15.78:0/4220264232 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147485738 0 0) 0x7f52340a2110 con 0x7f5234137ce0 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.747 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient(hunting): my global_id is 154142 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f521c003530 con 0x7f5234137ce0 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3385905852 0 0) 0x7f52340a6420 con 0x7f52340a9610 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient(hunting): my global_id is 144219 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52340a2110 con 0x7f52340a9610 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1958554277 0 0) 0x7f521c003530 con 0x7f5234137ce0 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 >> v1:172.21.15.146:6789/0 conn(0x7f523413e5f0 legacy=0x7f52341409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 >> v1:172.21.15.78:6789/0 conn(0x7f52340a9610 legacy=0x7f523413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.748 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: _finish_hunting 0 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 monclient: found mon.noname-b 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52340a4bc0 con 0x7f5234137ce0 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: _finish_auth 0 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: _check_auth_tickets 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f522c001a20 con 0x7f5234137ce0 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 5 monclient: authenticate success, global_id 154142 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: dump: 2024-09-15T20:48:49.749 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:49.750 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f522c001bc0 con 0x7f5234137ce0 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 1 -- 172.21.15.78:0/4220264232 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f522c001f90 con 0x7f5234137ce0 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 4 set_mon_vals no callback set 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f52389d7640 10 monclient: dump: 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.751 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: shutdown 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- 172.21.15.78:0/4220264232 >> v1:172.21.15.78:6790/0 conn(0x7f5234137ce0 legacy=0x7f52341380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- 172.21.15.78:0/4220264232 shutdown_connections 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- 172.21.15.78:0/4220264232 >> 172.21.15.78:0/4220264232 conn(0x7f52340aa6c0 msgr2=0x7f52340aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- 172.21.15.78:0/4220264232 shutdown_connections 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 1 -- 172.21.15.78:0/4220264232 wait complete. 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: build_initial_monmap 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.735+0000 7f523bc3d640 10 monclient: monmap: 2024-09-15T20:48:49.752 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 librados: starting msgr at 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 librados: starting objecter 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 Processor -- start 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 -- start start 2024-09-15T20:48:49.753 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 librados: setting wanted keys 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 librados: calling monclient init 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient: init 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient(hunting): picked mon.noname-a con 0x7f52340a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient(hunting): picked mon.noname-b con 0x7f5234137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient(hunting): picked mon.noname-c con 0x7f523413e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5234139cd0 con 0x7f52340a9610 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f523413c6f0 con 0x7f5234137ce0 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5234074310 con 0x7f523413e5f0 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient(hunting): authenticate will time out at 2190.624829s 2024-09-15T20:48:49.754 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52399d9640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f52340a9610 0x7f5234138eb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35194/0 (socket says 172.21.15.78:35194) 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52399d9640 1 -- 172.21.15.78:0/2603783631 learned_addr learned my addr 172.21.15.78:0/2603783631 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.?.objecter ms_handle_connect 0x7f523413e5f0 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.?.objecter ms_handle_connect 0x7f52340a9610 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2491643201 0 0) 0x7f5234074310 con 0x7f523413e5f0 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): my global_id is 154145 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5220004ec0 con 0x7f523413e5f0 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 910959687 0 0) 0x7f5234139cd0 con 0x7f52340a9610 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.755 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): my global_id is 144222 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5234074310 con 0x7f52340a9610 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.?.objecter ms_handle_connect 0x7f5234137ce0 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2269801838 0 0) 0x7f5220004ec0 con 0x7f523413e5f0 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5234139cd0 con 0x7f523413e5f0 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f52240018c0 con 0x7f523413e5f0 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): dump: 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.756 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2303703094 0 0) 0x7f5234074310 con 0x7f52340a9610 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5220004ec0 con 0x7f52340a9610 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f522c001f50 con 0x7f52340a9610 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:49.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): dump: 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 273605664 0 0) 0x7f523413c6f0 con 0x7f5234137ce0 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): my global_id is 134311 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5234074310 con 0x7f5234137ce0 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1971090694 0 0) 0x7f5234139cd0 con 0x7f523413e5f0 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 >> v1:172.21.15.146:6789/0 conn(0x7f5234137ce0 legacy=0x7f52341395c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 >> v1:172.21.15.78:6789/0 conn(0x7f52340a9610 legacy=0x7f5234138eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: _finish_hunting 0 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 monclient: found mon.c 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52340a91e0 con 0x7f523413e5f0 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: _finish_auth 0 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: _check_auth_tickets 2024-09-15T20:48:49.759 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 5 monclient: authenticate success, global_id 154145 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient: _renew_subs 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f523413c130 con 0x7f523413e5f0 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 client.154145.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient: _renew_subs 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f523413ce70 con 0x7f523413e5f0 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 librados: init done 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5224003600 con 0x7f523413e5f0 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5224003950 con 0x7f523413e5f0 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.760 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: dump: 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f5233fff640 4 set_mon_vals no callback set 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:49.761 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f52140065d0 con 0x7f5214002870 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f522400f9c0 con 0x7f523413e5f0 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.154145.objecter ms_handle_connect 0x7f5214002870 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.154145.objecter resend_mon_ops 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7f522404a590 con 0x7f523413e5f0 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.154145.objecter ms_dispatch 0x7f5234000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 3 client.154145.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 3 client.154145.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 20 client.154145.objecter dump_active .. 0 homeless 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1622845190 0 0) 0x7f52140065d0 con 0x7f5214002870 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient(hunting): my global_id is 144225 2024-09-15T20:48:49.762 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5234139cd0 con 0x7f5214002870 2024-09-15T20:48:49.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52391d8640 10 monclient: get_auth_request con 0x7f522003e2b0 auth_method 0 2024-09-15T20:48:49.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 224538364 0 0) 0x7f5234139cd0 con 0x7f5214002870 2024-09-15T20:48:49.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52140065d0 con 0x7f5214002870 2024-09-15T20:48:49.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f522c004e70 con 0x7f5214002870 2024-09-15T20:48:49.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.763 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: dump: 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.764 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 504715742 0 0) 0x7f52140065d0 con 0x7f5214002870 2024-09-15T20:48:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f521400a170 con 0x7f5214002870 2024-09-15T20:48:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 client.154145.objecter ms_handle_connect 0x7f522003e2b0 2024-09-15T20:48:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f522c0063d0 con 0x7f5214002870 2024-09-15T20:48:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 >> v1:172.21.15.78:6789/0 conn(0x7f5214002870 legacy=0x7f5214004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.765 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.739+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 >> v1:172.21.15.78:6789/0 conn(0x7f5214002870 legacy=0x7f5214004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:49.766 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f5214009e60 con 0x7f5214009120 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 client.154145.objecter ms_handle_connect 0x7f5214009120 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 client.154145.objecter resend_mon_ops 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 551298420 0 0) 0x7f5214009e60 con 0x7f5214009120 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient(hunting): my global_id is 144228 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52140065d0 con 0x7f5214009120 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4182330466 0 0) 0x7f52140065d0 con 0x7f5214009120 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5214009e60 con 0x7f5214009120 2024-09-15T20:48:49.767 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f522c005160 con 0x7f5214009120 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient: dump: 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.768 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 6749751 0 0) 0x7f5214009e60 con 0x7f5214009120 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f521400c400 con 0x7f5214009120 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4307 (unknown 0 0 3626395738) 0x7f522c006580 con 0x7f5214009120 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 >> v1:172.21.15.78:6789/0 conn(0x7f5214009120 legacy=0x7f52140095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f52327fc640 1 -- 172.21.15.78:0/2603783631 >> v1:172.21.15.78:6789/0 conn(0x7f5214009120 legacy=0x7f52140095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:48:49.769 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 70, 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 131642, 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:49.770 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:49.771 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:48:49.772 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:48:49.773 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.774 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.775 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:49.776 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:48:49.777 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:49.778 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:49.779 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 5 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 4 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.780 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:49.781 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:49.781 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:49.781 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.781 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.781 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.781 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:48:49.781 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:48:49.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 10 librados: watch_flush enter 2024-09-15T20:48:49.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 10 librados: watch_flush exit 2024-09-15T20:48:49.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 20 client.154145.objecter shutdown clearing up homeless session... 2024-09-15T20:48:49.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 10 client.154145.objecter successfully canceled tick 2024-09-15T20:48:49.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 >> v1:172.21.15.146:6812/435598642 conn(0x7f522003e2b0 legacy=0x7f5220040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 10 monclient: shutdown 2024-09-15T20:48:49.783 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:49.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 >> v1:172.21.15.78:6790/0 conn(0x7f523413e5f0 legacy=0x7f52341d4fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523a1da640 1 -- 172.21.15.78:0/2603783631 reap_dead start 2024-09-15T20:48:49.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523a1da640 1 -- 172.21.15.78:0/2603783631 reap_dead start 2024-09-15T20:48:49.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 shutdown_connections 2024-09-15T20:48:49.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 >> 172.21.15.78:0/2603783631 conn(0x7f52340aa6c0 msgr2=0x7f5234073a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:49.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 shutdown_connections 2024-09-15T20:48:49.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 1 -- 172.21.15.78:0/2603783631 wait complete. 2024-09-15T20:48:49.784 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.747+0000 7f523bc3d640 1 librados: shutdown 2024-09-15T20:48:49.784 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:48:49.784 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:48:49.785 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient: get_monmap_and_config 2024-09-15T20:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient: build_initial_monmap 2024-09-15T20:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient: monmap: 2024-09-15T20:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:49.902 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:49.905066+0000 2024-09-15T20:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:49.905066+0000 2024-09-15T20:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:49.903 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 Processor -- start 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- start start 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient: init 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient(hunting): picked mon.noname-a con 0x7f9c80135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient(hunting): picked mon.noname-b con 0x7f9c800a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient(hunting): picked mon.noname-c con 0x7f9c8013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:49.915 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c800745e0 con 0x7f9c80135b60 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c8006ecf0 con 0x7f9c8013c220 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c8006efd0 con 0x7f9c800a24e0 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient(hunting): authenticate will time out at 2190.789599s 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c85b5e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9c80135b60 0x7f9c80135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35226/0 (socket says 172.21.15.78:35226) 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c85b5e640 1 -- 172.21.15.78:0/1035460926 learned_addr learned my addr 172.21.15.78:0/1035460926 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3401762705 0 0) 0x7f9c8006efd0 con 0x7f9c800a24e0 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.916 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient(hunting): my global_id is 154148 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c6c003530 con 0x7f9c800a24e0 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1653615314 0 0) 0x7f9c800745e0 con 0x7f9c80135b60 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient(hunting): my global_id is 144231 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c8006efd0 con 0x7f9c80135b60 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3942173042 0 0) 0x7f9c6c003530 con 0x7f9c800a24e0 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 >> v1:172.21.15.146:6789/0 conn(0x7f9c8013c220 legacy=0x7f9c8013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 >> v1:172.21.15.78:6789/0 conn(0x7f9c80135b60 legacy=0x7f9c80135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.917 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: _finish_hunting 0 2024-09-15T20:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 monclient: found mon.noname-b 2024-09-15T20:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c80136650 con 0x7f9c800a24e0 2024-09-15T20:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: _finish_auth 0 2024-09-15T20:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: _check_auth_tickets 2024-09-15T20:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:49.918 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9c700016a0 con 0x7f9c800a24e0 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: dump: 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.919 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 5 monclient: authenticate success, global_id 154148 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c700018c0 con 0x7f9c800a24e0 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 4 set_mon_vals no callback set 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 1 -- 172.21.15.78:0/1035460926 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9c70001cd0 con 0x7f9c800a24e0 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c84b5c640 10 monclient: dump: 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.920 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 10 monclient: shutdown 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- 172.21.15.78:0/1035460926 >> v1:172.21.15.78:6790/0 conn(0x7f9c800a24e0 legacy=0x7f9c8013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- 172.21.15.78:0/1035460926 shutdown_connections 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- 172.21.15.78:0/1035460926 >> 172.21.15.78:0/1035460926 conn(0x7f9c800a63c0 msgr2=0x7f9c800a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- 172.21.15.78:0/1035460926 shutdown_connections 2024-09-15T20:48:49.921 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.903+0000 7f9c87dc2640 1 -- 172.21.15.78:0/1035460926 wait complete. 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: build_initial_monmap 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: monmap: 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:49.922 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 librados: starting msgr at 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 librados: starting objecter 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 Processor -- start 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 -- start start 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 librados: setting wanted keys 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 librados: calling monclient init 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: init 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient(hunting): picked mon.noname-b con 0x7f9c800a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:49.923 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient(hunting): picked mon.noname-c con 0x7f9c80135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient(hunting): picked mon.noname-a con 0x7f9c8013c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c80138320 con 0x7f9c8013c220 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c80072950 con 0x7f9c800a24e0 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c80138600 con 0x7f9c80135b60 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient(hunting): authenticate will time out at 2190.792918s 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c8535d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f9c80135b60 0x7f9c80072240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33966/0 (socket says 172.21.15.78:33966) 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c8635f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f9c8013c220 0x7f9c801d2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35232/0 (socket says 172.21.15.78:35232) 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c8535d640 1 -- 172.21.15.78:0/165282207 learned_addr learned my addr 172.21.15.78:0/165282207 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.?.objecter ms_handle_connect 0x7f9c8013c220 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.?.objecter ms_handle_connect 0x7f9c80135b60 2024-09-15T20:48:49.924 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1794550472 0 0) 0x7f9c80138320 con 0x7f9c8013c220 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): my global_id is 144234 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c64004ec0 con 0x7f9c8013c220 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2104144655 0 0) 0x7f9c80138600 con 0x7f9c80135b60 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): my global_id is 154151 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c80138320 con 0x7f9c80135b60 2024-09-15T20:48:49.925 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.?.objecter ms_handle_connect 0x7f9c800a24e0 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2987139003 0 0) 0x7f9c64004ec0 con 0x7f9c8013c220 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c80138600 con 0x7f9c8013c220 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9c78001c80 con 0x7f9c8013c220 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): dump: 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.926 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4031416695 0 0) 0x7f9c80138320 con 0x7f9c80135b60 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c64004ec0 con 0x7f9c80135b60 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2000316945 0 0) 0x7f9c80072950 con 0x7f9c800a24e0 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.927 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): my global_id is 134317 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c80138320 con 0x7f9c800a24e0 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9c70001c10 con 0x7f9c80135b60 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): dump: 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.928 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 905753368 0 0) 0x7f9c80138600 con 0x7f9c8013c220 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 >> v1:172.21.15.78:6790/0 conn(0x7f9c80135b60 legacy=0x7f9c80072240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 >> v1:172.21.15.146:6789/0 conn(0x7f9c800a24e0 legacy=0x7f9c80071b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: _finish_hunting 0 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 monclient: found mon.a 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:49.929 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8013f8e0 con 0x7f9c8013c220 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: _finish_auth 0 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: _check_auth_tickets 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 5 monclient: authenticate success, global_id 144234 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: _renew_subs 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f9c8013ba20 con 0x7f9c8013c220 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 client.144234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: _renew_subs 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f9c8013b540 con 0x7f9c8013c220 2024-09-15T20:48:49.930 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 librados: init done 2024-09-15T20:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9c780039a0 con 0x7f9c8013c220 2024-09-15T20:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9c78003d30 con 0x7f9c8013c220 2024-09-15T20:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: dump: 2024-09-15T20:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.931 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.932 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c777fe640 4 set_mon_vals no callback set 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9c680065d0 con 0x7f9c68002870 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.144234.objecter ms_handle_connect 0x7f9c68002870 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.144234.objecter resend_mon_ops 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f9c78003ed0 con 0x7f9c8013c220 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7f9c7804baf0 con 0x7f9c8013c220 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.144234.objecter ms_dispatch 0x7f9c80000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 3 client.144234.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 3 client.144234.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 20 client.144234.objecter dump_active .. 0 homeless 2024-09-15T20:48:49.933 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3896559513 0 0) 0x7f9c680065d0 con 0x7f9c68002870 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient(hunting): my global_id is 154154 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c80138600 con 0x7f9c68002870 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c8535d640 10 monclient: get_auth_request con 0x7f9c640426d0 auth_method 0 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 811349203 0 0) 0x7f9c80138600 con 0x7f9c68002870 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c680065d0 con 0x7f9c68002870 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9c7c001950 con 0x7f9c68002870 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 monclient: dump: 2024-09-15T20:48:49.934 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 2618734134 0 0) 0x7f9c680065d0 con 0x7f9c68002870 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9c6800a170 con 0x7f9c68002870 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.907+0000 7f9c767fc640 10 client.144234.objecter ms_handle_connect 0x7f9c640426d0 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.911+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f9c7c001a20 con 0x7f9c68002870 2024-09-15T20:48:49.935 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.911+0000 7f9c767fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.911+0000 7f9c767fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.911+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 >> v1:172.21.15.78:6790/0 conn(0x7f9c68002870 legacy=0x7f9c68004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.936 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.911+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 >> v1:172.21.15.78:6790/0 conn(0x7f9c68002870 legacy=0x7f9c68004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9c68009e80 con 0x7f9c68009140 2024-09-15T20:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 client.144234.objecter ms_handle_connect 0x7f9c68009140 2024-09-15T20:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 client.144234.objecter resend_mon_ops 2024-09-15T20:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2559780646 0 0) 0x7f9c68009e80 con 0x7f9c68009140 2024-09-15T20:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient(hunting): _negotiate 2024-09-15T20:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:49.937 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient(hunting): my global_id is 154157 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c680065d0 con 0x7f9c68009140 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3616783984 0 0) 0x7f9c680065d0 con 0x7f9c68009140 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9c68009e80 con 0x7f9c68009140 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9c7c004990 con 0x7f9c68009140 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient: dump: 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:49.938 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 231941981 0 0) 0x7f9c68009e80 con 0x7f9c68009140 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9c6800c400 con 0x7f9c68009140 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3970 (unknown 0 0 1154890232) 0x7f9c7c004920 con 0x7f9c68009140 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 >> v1:172.21.15.78:6790/0 conn(0x7f9c68009140 legacy=0x7f9c680095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.939 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c767fc640 1 -- 172.21.15.78:0/165282207 >> v1:172.21.15.78:6790/0 conn(0x7f9c68009140 legacy=0x7f9c680095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 70, 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 17899, 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:48:49.940 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 2, 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:49.941 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:49.942 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:48:49.943 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:49.944 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:49.945 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:49.946 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:49.947 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.948 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:48:49.949 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.950 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:48:49.951 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 10 librados: watch_flush enter 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 10 librados: watch_flush exit 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 20 client.144234.objecter shutdown clearing up homeless session... 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 10 client.144234.objecter successfully canceled tick 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 >> v1:172.21.15.146:6812/435598642 conn(0x7f9c640426d0 legacy=0x7f9c64044b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 10 monclient: shutdown 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 >> v1:172.21.15.78:6789/0 conn(0x7f9c8013c220 legacy=0x7f9c801d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:49.953 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c8635f640 1 -- 172.21.15.78:0/165282207 reap_dead start 2024-09-15T20:48:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c8635f640 1 -- 172.21.15.78:0/165282207 reap_dead start 2024-09-15T20:48:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 shutdown_connections 2024-09-15T20:48:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 >> 172.21.15.78:0/165282207 conn(0x7f9c800a63c0 msgr2=0x7f9c8012eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 shutdown_connections 2024-09-15T20:48:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 1 -- 172.21.15.78:0/165282207 wait complete. 2024-09-15T20:48:49.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:49.915+0000 7f9c87dc2640 1 librados: shutdown 2024-09-15T20:48:49.954 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:48:49.954 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:48:49.954 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:48:50.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient: get_monmap_and_config 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient: build_initial_monmap 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient: monmap: 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:50.073563+0000 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:50.073563+0000 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:50.071 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 Processor -- start 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- start start 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient: init 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient(hunting): picked mon.noname-a con 0x7f3748137ce0 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient(hunting): picked mon.noname-c con 0x7f37480a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient(hunting): picked mon.noname-b con 0x7f374813e5f0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:50.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37480a6420 con 0x7f3748137ce0 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37480a6700 con 0x7f37480a9610 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37480a2110 con 0x7f374813e5f0 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient(hunting): authenticate will time out at 2190.958039s 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f37477fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f374813e5f0 0x7f37481409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33982/0 (socket says 172.21.15.78:33982) 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3746ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3748137ce0 0x7f37481380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35238/0 (socket says 172.21.15.78:35238) 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f37477fe640 1 -- 172.21.15.78:0/996603060 learned_addr learned my addr 172.21.15.78:0/996603060 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2510860582 0 0) 0x7f37480a6420 con 0x7f3748137ce0 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient(hunting): my global_id is 144237 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3734003530 con 0x7f3748137ce0 2024-09-15T20:48:50.081 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 632980128 0 0) 0x7f37480a2110 con 0x7f374813e5f0 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient(hunting): my global_id is 154160 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37480a6420 con 0x7f374813e5f0 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3970513148 0 0) 0x7f3734003530 con 0x7f3748137ce0 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 >> v1:172.21.15.78:6790/0 conn(0x7f374813e5f0 legacy=0x7f37481409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 >> v1:172.21.15.146:6789/0 conn(0x7f37480a9610 legacy=0x7f374813ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 monclient: found mon.noname-a 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37480a4bc0 con 0x7f3748137ce0 2024-09-15T20:48:50.082 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: _finish_auth 0 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f37380016a0 con 0x7f3748137ce0 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: dump: 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.083 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 5 monclient: authenticate success, global_id 144237 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f37380018c0 con 0x7f3748137ce0 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 4 set_mon_vals no callback set 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 1 -- 172.21.15.78:0/996603060 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3738001c90 con 0x7f3748137ce0 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f3745ffb640 10 monclient: dump: 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.084 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 10 monclient: shutdown 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- 172.21.15.78:0/996603060 >> v1:172.21.15.78:6789/0 conn(0x7f3748137ce0 legacy=0x7f37481380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- 172.21.15.78:0/996603060 shutdown_connections 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- 172.21.15.78:0/996603060 >> 172.21.15.78:0/996603060 conn(0x7f37480aa6c0 msgr2=0x7f37480aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:50.085 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- 172.21.15.78:0/996603060 shutdown_connections 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.071+0000 7f374d33c640 1 -- 172.21.15.78:0/996603060 wait complete. 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: build_initial_monmap 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: monmap: 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.086 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 librados: starting msgr at 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 librados: starting objecter 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 Processor -- start 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 -- start start 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 librados: setting wanted keys 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 librados: calling monclient init 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: init 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient(hunting): picked mon.noname-a con 0x7f37480a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient(hunting): picked mon.noname-c con 0x7f3748137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient(hunting): picked mon.noname-b con 0x7f374813e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374813e3a0 con 0x7f37480a9610 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374809e670 con 0x7f374813e5f0 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f374809f1d0 con 0x7f3748137ce0 2024-09-15T20:48:50.087 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient(hunting): authenticate will time out at 2190.961035s 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f37467fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f3748137ce0 0x7f374809d5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:33986/0 (socket says 172.21.15.78:33986) 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f3746ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f37480a9610 0x7f374809ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:35242/0 (socket says 172.21.15.78:35242) 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f37467fc640 1 -- 172.21.15.78:0/3213173939 learned_addr learned my addr 172.21.15.78:0/3213173939 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.?.objecter ms_handle_connect 0x7f3748137ce0 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.?.objecter ms_handle_connect 0x7f37480a9610 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1307641993 0 0) 0x7f374809f1d0 con 0x7f3748137ce0 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.088 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): my global_id is 154163 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3728004ec0 con 0x7f3748137ce0 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2152219436 0 0) 0x7f374813e3a0 con 0x7f37480a9610 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): my global_id is 144240 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f374809f1d0 con 0x7f37480a9610 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.?.objecter ms_handle_connect 0x7f374813e5f0 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1586330972 0 0) 0x7f3728004ec0 con 0x7f3748137ce0 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f374813e3a0 con 0x7f3748137ce0 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2337657418 0 0) 0x7f374809f1d0 con 0x7f37480a9610 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3728004ec0 con 0x7f37480a9610 2024-09-15T20:48:50.089 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f37300019e0 con 0x7f3748137ce0 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): dump: 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:50.090 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3738001c10 con 0x7f37480a9610 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): dump: 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.091 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1292893563 0 0) 0x7f374809e670 con 0x7f374813e5f0 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): my global_id is 134323 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f374809f1d0 con 0x7f374813e5f0 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 771753923 0 0) 0x7f374813e3a0 con 0x7f3748137ce0 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 >> v1:172.21.15.146:6789/0 conn(0x7f374813e5f0 legacy=0x7f37481d4f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 >> v1:172.21.15.78:6789/0 conn(0x7f37480a9610 legacy=0x7f374809ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 monclient: found mon.c 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:50.092 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f374809ef20 con 0x7f3748137ce0 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient: _finish_auth 0 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 5 monclient: authenticate success, global_id 154163 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: _renew_subs 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f374809fd90 con 0x7f3748137ce0 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3730001d20 con 0x7f3748137ce0 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 client.154163.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3730003a70 con 0x7f3748137ce0 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: _renew_subs 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:48:50.093 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f374809dec0 con 0x7f3748137ce0 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 librados: init done 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f37457fa640 4 set_mon_vals no callback set 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient: dump: 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.094 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f37480a14b0 con 0x7f374813e9d0 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f3730003c10 con 0x7f3748137ce0 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7f373004b810 con 0x7f3748137ce0 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.154163.objecter ms_dispatch 0x7f3748000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 3 client.154163.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 3 client.154163.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 20 client.154163.objecter dump_active .. 0 homeless 2024-09-15T20:48:50.095 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.154163.objecter ms_handle_connect 0x7f374813e9d0 2024-09-15T20:48:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.154163.objecter resend_mon_ops 2024-09-15T20:48:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f37477fe640 10 monclient: get_auth_request con 0x7f372803e2b0 auth_method 0 2024-09-15T20:48:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004355142 0 0) 0x7f37480a14b0 con 0x7f374813e9d0 2024-09-15T20:48:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 monclient(hunting): my global_id is 134326 2024-09-15T20:48:50.096 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f374813e3a0 con 0x7f374813e9d0 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.075+0000 7f373f7fe640 10 client.154163.objecter ms_handle_connect 0x7f372803e2b0 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4139692386 0 0) 0x7f374813e3a0 con 0x7f374813e9d0 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f37480a14b0 con 0x7f374813e9d0 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3738004ce0 con 0x7f374813e9d0 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 10 monclient: dump: 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.097 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3065902416 0 0) 0x7f37480a14b0 con 0x7f374813e9d0 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f374806a080 con 0x7f374813e9d0 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f37380061c0 con 0x7f374813e9d0 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 >> v1:172.21.15.146:6789/0 conn(0x7f374813e9d0 legacy=0x7f374813edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.098 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.079+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 >> v1:172.21.15.146:6789/0 conn(0x7f374813e9d0 legacy=0x7f374813edb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.083+0000 7f374d33c640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.083+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f3748074e80 con 0x7f374813f4c0 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 client.154163.objecter ms_handle_connect 0x7f374813f4c0 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 client.154163.objecter resend_mon_ops 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3892795190 0 0) 0x7f3748074e80 con 0x7f374813f4c0 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient(hunting): my global_id is 134329 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f37480a14b0 con 0x7f374813f4c0 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 103118588 0 0) 0x7f37480a14b0 con 0x7f374813f4c0 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3748074e80 con 0x7f374813f4c0 2024-09-15T20:48:50.100 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3738006800 con 0x7f374813f4c0 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient: dump: 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3734992173 0 0) 0x7f3748074e80 con 0x7f374813f4c0 2024-09-15T20:48:50.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3748076db0 con 0x7f374813f4c0 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4138 (unknown 0 0 924396641) 0x7f3738006620 con 0x7f374813f4c0 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 >> v1:172.21.15.146:6789/0 conn(0x7f374813f4c0 legacy=0x7f37481da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f373f7fe640 1 -- 172.21.15.78:0/3213173939 >> v1:172.21.15.146:6789/0 conn(0x7f374813f4c0 legacy=0x7f37481da690 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 70, 2024-09-15T20:48:50.102 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 239955, 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 3, 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:50.103 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:50.104 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:48:50.105 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:48:50.106 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.107 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:48:50.108 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:50.109 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.110 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.111 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:50.112 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 3 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:48:50.113 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:48:50.114 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:48:50.114 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 10 librados: watch_flush enter 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 10 librados: watch_flush exit 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 20 client.154163.objecter shutdown clearing up homeless session... 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 10 client.154163.objecter successfully canceled tick 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 >> v1:172.21.15.146:6812/435598642 conn(0x7f372803e2b0 legacy=0x7f3728040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 10 monclient: shutdown 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 >> v1:172.21.15.78:6790/0 conn(0x7f3748137ce0 legacy=0x7f374809d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f37477fe640 1 -- 172.21.15.78:0/3213173939 reap_dead start 2024-09-15T20:48:50.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f37477fe640 1 -- 172.21.15.78:0/3213173939 reap_dead start 2024-09-15T20:48:50.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 shutdown_connections 2024-09-15T20:48:50.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 >> 172.21.15.78:0/3213173939 conn(0x7f37480aa6c0 msgr2=0x7f37480a2f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:50.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 shutdown_connections 2024-09-15T20:48:50.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 1 -- 172.21.15.78:0/3213173939 wait complete. 2024-09-15T20:48:50.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.087+0000 7f374d33c640 1 librados: shutdown 2024-09-15T20:48:50.120 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:48:50.120 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-09-15T20:48:50.120 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-09-15T20:48:50.121 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-09-15T20:48:50.121 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-09-15T20:48:50.121 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T20:48:50.121 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:48:50.146 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T20:48:50.146 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-09-15T20:48:50.146 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:48:50.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient: get_monmap_and_config 2024-09-15T20:48:50.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient: build_initial_monmap 2024-09-15T20:48:50.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient: monmap: 2024-09-15T20:48:50.260 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:50.260 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.261 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:50.263028+0000 2024-09-15T20:48:50.261 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:50.263028+0000 2024-09-15T20:48:50.261 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:50.261 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.261 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:50.261 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:50.261 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:50.261 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 1 Processor -- start 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 1 -- start start 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient: init 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient(hunting): picked mon.noname-a con 0x7fc610135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient(hunting): picked mon.noname-b con 0x7fc6100a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient(hunting): picked mon.noname-c con 0x7fc61013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6100745e0 con 0x7fc610135b60 2024-09-15T20:48:50.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc61006ecf0 con 0x7fc61013c220 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc61006efd0 con 0x7fc6100a24e0 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc6187a7640 10 monclient(hunting): authenticate will time out at 2191.147505s 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc615d42640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc6100a24e0 0x7fc61013ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:60806/0 (socket says 172.21.15.78:60806) 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc615d42640 1 -- 172.21.15.78:0/4068036281 learned_addr learned my addr 172.21.15.78:0/4068036281 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc616d44640 1 -- 172.21.15.78:0/4068036281 >> v1:172.21.15.146:6789/0 conn(0x7fc61013c220 legacy=0x7fc61013e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.259+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2876720791 0 0) 0x7fc6100745e0 con 0x7fc610135b60 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient(hunting): my global_id is 144243 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f8003530 con 0x7fc610135b60 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 551035424 0 0) 0x7fc5f8003530 con 0x7fc610135b60 2024-09-15T20:48:50.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 >> v1:172.21.15.78:6790/0 conn(0x7fc6100a24e0 legacy=0x7fc61013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 >> v1:172.21.15.146:6789/0 conn(0x7fc61013c220 legacy=0x7fc61013e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: _finish_hunting 0 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 monclient: found mon.noname-a 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc610136650 con 0x7fc610135b60 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: _finish_auth 0 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: _check_auth_tickets 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc6080016a0 con 0x7fc610135b60 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: dump: 2024-09-15T20:48:50.270 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 5 monclient: authenticate success, global_id 144243 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6080018c0 con 0x7fc610135b60 2024-09-15T20:48:50.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 1 -- 172.21.15.78:0/4068036281 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc608001cd0 con 0x7fc610135b60 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 4 set_mon_vals no callback set 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc615541640 10 monclient: dump: 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.272 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: shutdown 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- 172.21.15.78:0/4068036281 >> v1:172.21.15.78:6789/0 conn(0x7fc610135b60 legacy=0x7fc610135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- 172.21.15.78:0/4068036281 shutdown_connections 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- 172.21.15.78:0/4068036281 >> 172.21.15.78:0/4068036281 conn(0x7fc6100a63c0 msgr2=0x7fc6100a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- 172.21.15.78:0/4068036281 shutdown_connections 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- 172.21.15.78:0/4068036281 wait complete. 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: build_initial_monmap 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: monmap: 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:50.273 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 librados: starting msgr at 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 librados: starting objecter 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 Processor -- start 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- start start 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 librados: setting wanted keys 2024-09-15T20:48:50.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 librados: calling monclient init 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: init 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient(hunting): picked mon.noname-b con 0x7fc6100a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient(hunting): picked mon.noname-a con 0x7fc610135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient(hunting): picked mon.noname-c con 0x7fc61013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc610074d70 con 0x7fc610135b60 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc610138790 con 0x7fc6100a24e0 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc610073710 con 0x7fc61013c220 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:50.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient(hunting): authenticate will time out at 2191.150372s 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc616d44640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc61013c220 0x7fc6101d2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:60816/0 (socket says 172.21.15.78:60816) 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc616d44640 1 -- 172.21.15.78:0/1013633529 learned_addr learned my addr 172.21.15.78:0/1013633529 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc616543640 1 -- 172.21.15.78:0/1013633529 >> v1:172.21.15.146:6789/0 conn(0x7fc6100a24e0 legacy=0x7fc610071f00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 client.?.objecter ms_handle_connect 0x7fc61013c220 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 client.?.objecter ms_handle_connect 0x7fc610135b60 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3752038140 0 0) 0x7fc61006efd0 con 0x7fc610135b60 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient(hunting): _negotiate 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient(hunting): my global_id is 144246 2024-09-15T20:48:50.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5fc004ef0 con 0x7fc610135b60 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3804054754 0 0) 0x7fc5fc004ef0 con 0x7fc610135b60 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc61006efd0 con 0x7fc610135b60 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc604001870 con 0x7fc610135b60 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient(hunting): dump: 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.277 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2538324883 0 0) 0x7fc61006efd0 con 0x7fc610135b60 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 >> v1:172.21.15.78:6790/0 conn(0x7fc61013c220 legacy=0x7fc6101d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 >> v1:172.21.15.146:6789/0 conn(0x7fc6100a24e0 legacy=0x7fc610071f00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient: _finish_hunting 0 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 monclient: found mon.a 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc610137ef0 con 0x7fc610135b60 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient: _finish_auth 0 2024-09-15T20:48:50.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient: _check_auth_tickets 2024-09-15T20:48:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 5 monclient: authenticate success, global_id 144246 2024-09-15T20:48:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: _renew_subs 2024-09-15T20:48:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc610137ba0 con 0x7fc610135b60 2024-09-15T20:48:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 client.144246.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:50.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: _renew_subs 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc610072d20 con 0x7fc610135b60 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc6187a7640 1 librados: init done 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc6040047f0 con 0x7fc610135b60 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc604004c00 con 0x7fc610135b60 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc614d40640 4 set_mon_vals no callback set 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.263+0000 7fc60effd640 10 monclient: dump: 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.280 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc6187a7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc6187a7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc6187a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5f0005090 con 0x7fc610135b60 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fc604004da0 con 0x7fc610135b60 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7fc60404b800 con 0x7fc610135b60 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc60effd640 10 client.144246.objecter ms_dispatch 0x7fc610000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:48:50.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc60effd640 3 client.144246.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:48:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc60effd640 3 client.144246.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:48:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc60effd640 20 client.144246.objecter dump_active .. 0 homeless 2024-09-15T20:48:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc616543640 10 monclient: get_auth_request con 0x7fc5fc03df80 auth_method 0 2024-09-15T20:48:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.267+0000 7fc60effd640 10 client.144246.objecter ms_handle_connect 0x7fc5fc03df80 2024-09-15T20:48:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.271+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fc610074d70 con 0x7fc610135b60 2024-09-15T20:48:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.271+0000 7fc60effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:50.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.271+0000 7fc60effd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:50.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:48:50.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:48:50.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:50.499 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc5f0005380 con 0x7fc610135b60 2024-09-15T20:48:50.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc60effd640 1 -- 172.21.15.78:0/1013633529 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 109566940) 0x7fc604016a70 con 0x7fc610135b60 2024-09-15T20:48:50.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc60effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:48:50.500 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc60effd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:50.500 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:48:50.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 10 librados: watch_flush enter 2024-09-15T20:48:50.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 10 librados: watch_flush exit 2024-09-15T20:48:50.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 20 client.144246.objecter shutdown clearing up homeless session... 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 10 client.144246.objecter successfully canceled tick 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.499+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 >> v1:172.21.15.146:6812/435598642 conn(0x7fc5fc03df80 legacy=0x7fc5fc040440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.503+0000 7fc6187a7640 10 monclient: shutdown 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.503+0000 7fc6187a7640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.503+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 >> v1:172.21.15.78:6789/0 conn(0x7fc610135b60 legacy=0x7fc610072610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.503+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 shutdown_connections 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.503+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 >> 172.21.15.78:0/1013633529 conn(0x7fc6100a63c0 msgr2=0x7fc6100a3e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.503+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 shutdown_connections 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.503+0000 7fc6187a7640 1 -- 172.21.15.78:0/1013633529 wait complete. 2024-09-15T20:48:50.503 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:50.503+0000 7fc6187a7640 1 librados: shutdown 2024-09-15T20:48:53.518 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:48:53.645 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.643+0000 7f057d8c2640 10 monclient: get_monmap_and_config 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.643+0000 7f057d8c2640 10 monclient: build_initial_monmap 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.643+0000 7f057d8c2640 10 monclient: monmap: 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:53.648353+0000 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:53.648353+0000 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:53.646 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 Processor -- start 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- start start 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: init 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient(hunting): picked mon.noname-b con 0x7f0578135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient(hunting): picked mon.noname-c con 0x7f05780a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient(hunting): picked mon.noname-a con 0x7f057813c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05780745e0 con 0x7f057813c220 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057806ecf0 con 0x7f05780a24e0 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057806efd0 con 0x7f0578135b60 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:53.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient(hunting): authenticate will time out at 2194.532927s 2024-09-15T20:48:53.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0576ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f0578135b60 0x7f0578135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:60820/0 (socket says 172.21.15.78:60820) 2024-09-15T20:48:53.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0576ffd640 1 -- 172.21.15.78:0/1988974131 learned_addr learned my addr 172.21.15.78:0/1988974131 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:53.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f05777fe640 1 --1- 172.21.15.78:0/1988974131 >> v1:172.21.15.78:6789/0 conn(0x7f057813c220 0x7f057813e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49002/0 (socket says 172.21.15.78:49002) 2024-09-15T20:48:53.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f05767fc640 1 -- 172.21.15.78:0/1988974131 >> v1:172.21.15.146:6789/0 conn(0x7f05780a24e0 legacy=0x7f057813ab00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:48:53.654 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3773106021 0 0) 0x7f05780745e0 con 0x7f057813c220 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient(hunting): my global_id is 144249 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0560003530 con 0x7f057813c220 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 309712907 0 0) 0x7f0560003530 con 0x7f057813c220 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 >> v1:172.21.15.78:6790/0 conn(0x7f0578135b60 legacy=0x7f0578135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 >> v1:172.21.15.146:6789/0 conn(0x7f05780a24e0 legacy=0x7f057813ab00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 monclient: found mon.noname-a 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:48:53.655 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0578136650 con 0x7f057813c220 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: _finish_auth 0 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 5 monclient: authenticate success, global_id 144249 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0570001a20 con 0x7f057813c220 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: dump: 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:53.656 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0570001c40 con 0x7f057813c220 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: get_monmap_and_config success 2024-09-15T20:48:53.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 4 set_mon_vals no callback set 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 1 -- 172.21.15.78:0/1988974131 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0570002050 con 0x7f057813c220 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f0575ffb640 10 monclient: dump: 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:53.658 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: shutdown 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- 172.21.15.78:0/1988974131 >> v1:172.21.15.78:6789/0 conn(0x7f057813c220 legacy=0x7f057813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- 172.21.15.78:0/1988974131 shutdown_connections 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- 172.21.15.78:0/1988974131 >> 172.21.15.78:0/1988974131 conn(0x7f05780a63c0 msgr2=0x7f05780a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- 172.21.15.78:0/1988974131 shutdown_connections 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- 172.21.15.78:0/1988974131 wait complete. 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: build_initial_monmap 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: monmap: 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:48:53.659 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 librados: starting msgr at 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 librados: starting objecter 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 Processor -- start 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 -- start start 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 librados: setting wanted keys 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 1 librados: calling monclient init 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: init 2024-09-15T20:48:53.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.647+0000 7f057d8c2640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient(hunting): picked mon.noname-b con 0x7f05780a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient(hunting): picked mon.noname-a con 0x7f0578135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient(hunting): picked mon.noname-c con 0x7f057813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057809e690 con 0x7f0578135b60 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05780729e0 con 0x7f05780a24e0 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f057813fa30 con 0x7f057813c220 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient(hunting): authenticate will time out at 2194.536072s 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f05767fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0578135b60 0x7f05780722d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:49004/0 (socket says 172.21.15.78:49004) 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f05767fc640 1 -- 172.21.15.78:0/1102446626 learned_addr learned my addr 172.21.15.78:0/1102446626 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f05777fe640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f057813c220 0x7f05781d4e40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:60824/0 (socket says 172.21.15.78:60824) 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f0576ffd640 1 -- 172.21.15.78:0/1102446626 >> v1:172.21.15.146:6789/0 conn(0x7f05780a24e0 legacy=0x7f0578071bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:48:53.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 client.?.objecter ms_handle_connect 0x7f0578135b60 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 client.?.objecter ms_handle_connect 0x7f057813c220 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806425979 0 0) 0x7f057809e690 con 0x7f0578135b60 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient(hunting): my global_id is 144252 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0564004ec0 con 0x7f0578135b60 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1798738361 0 0) 0x7f0564004ec0 con 0x7f0578135b60 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f057809e690 con 0x7f0578135b60 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f056c004210 con 0x7f0578135b60 2024-09-15T20:48:53.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient(hunting): dump: 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:48:53.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1166215758 0 0) 0x7f057809e690 con 0x7f0578135b60 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 >> v1:172.21.15.78:6790/0 conn(0x7f057813c220 legacy=0x7f05781d4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 >> v1:172.21.15.146:6789/0 conn(0x7f05780a24e0 legacy=0x7f0578071bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 monclient: found mon.a 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f057809d180 con 0x7f0578135b60 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient: _finish_auth 0 2024-09-15T20:48:53.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 5 monclient: authenticate success, global_id 144252 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient: _renew_subs 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f057809e3b0 con 0x7f0578135b60 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 client.144252.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient: _renew_subs 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f057813af40 con 0x7f0578135b60 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 1 librados: init done 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f056c004430 con 0x7f0578135b60 2024-09-15T20:48:53.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f056c004840 con 0x7f0578135b60 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 monclient: dump: 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:48:53.666 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f05757fa640 4 set_mon_vals no callback set 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0558005090 con 0x7f0578135b60 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f056c0049e0 con 0x7f0578135b60 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7f056c04b870 con 0x7f0578135b60 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 client.144252.objecter ms_dispatch 0x7f0578000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 3 client.144252.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 3 client.144252.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 20 client.144252.objecter dump_active .. 0 homeless 2024-09-15T20:48:53.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f0576ffd640 10 monclient: get_auth_request con 0x7f056403df70 auth_method 0 2024-09-15T20:48:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.651+0000 7f055ffff640 10 client.144252.objecter ms_handle_connect 0x7f056403df70 2024-09-15T20:48:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.655+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f056c016aa0 con 0x7f0578135b60 2024-09-15T20:48:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.655+0000 7f055ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:48:53.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.655+0000 7f055ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:48:53.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.887+0000 7f057d8c2640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:48:53.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.887+0000 7f057d8c2640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:48:53.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.887+0000 7f057d8c2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:48:53.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.887+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0558005380 con 0x7f0578135b60 2024-09-15T20:48:53.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f055ffff640 1 -- 172.21.15.78:0/1102446626 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 18463429) 0x7f056c0163a0 con 0x7f0578135b60 2024-09-15T20:48:53.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f055ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:48:53.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f055ffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:48:53.891 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":70,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:48:53.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 10 librados: watch_flush enter 2024-09-15T20:48:53.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 10 librados: watch_flush exit 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 20 client.144252.objecter shutdown clearing up homeless session... 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 10 client.144252.objecter successfully canceled tick 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 >> v1:172.21.15.146:6812/435598642 conn(0x7f056403df70 legacy=0x7f0564040430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 10 monclient: shutdown 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 >> v1:172.21.15.78:6789/0 conn(0x7f0578135b60 legacy=0x7f05780722d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 shutdown_connections 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 >> 172.21.15.78:0/1102446626 conn(0x7f05780a63c0 msgr2=0x7f057812eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:48:53.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 shutdown_connections 2024-09-15T20:48:53.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 1 -- 172.21.15.78:0/1102446626 wait complete. 2024-09-15T20:48:53.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:53.891+0000 7f057d8c2640 1 librados: shutdown 2024-09-15T20:48:56.910 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:48:57.036 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient: get_monmap_and_config 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient: build_initial_monmap 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient: monmap: 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:48:57.039028+0000 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:48:57.039028+0000 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:48:57.037 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:48:57.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 1 Processor -- start 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 1 -- start start 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient: init 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient: _reopen_session rank -1 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient(hunting): picked mon.noname-a con 0x7f4bac135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient(hunting): picked mon.noname-b con 0x7f4bac0a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient(hunting): picked mon.noname-c con 0x7f4bac13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac0745e0 con 0x7f4bac135b60 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac06ecf0 con 0x7f4bac13c220 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac06efd0 con 0x7f4bac0a24e0 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient(hunting): _renew_subs 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4bb0b67640 10 monclient(hunting): authenticate will time out at 2197.923634s 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4baad9d640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4bac13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:48:57.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4ba9d9b640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f4bac0a24e0 0x7f4bac13ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:60838/0 (socket says 172.21.15.78:60838) 2024-09-15T20:48:57.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.035+0000 7f4ba9d9b640 1 -- 172.21.15.78:0/3346881759 learned_addr learned my addr 172.21.15.78:0/3346881759 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:48:57.239 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.239+0000 7f4baad9d640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4bac13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:48:57.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:57.643+0000 7f4baad9d640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4bac13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:48:58.447 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:48:58.447+0000 7f4baad9d640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4bac13e610 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 10 monclient(hunting): tick 2024-09-15T20:49:00.038 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 1 monclient(hunting): continuing hunt 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.78:6790/0 conn(0x7f4bac0a24e0 legacy=0x7f4bac13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4bac13e610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.78:6789/0 conn(0x7f4bac135b60 legacy=0x7f4bac135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f4b8c0036d0 addr v1:172.21.15.78:6790/0 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.035+0000 7f4b9bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f4bac135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.039+0000 7f4b9bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f4bac13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac136650 con 0x7f4bac135b60 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac06ecf0 con 0x7f4bac13c220 2024-09-15T20:49:00.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b8c010400 con 0x7f4b8c0036d0 2024-09-15T20:49:00.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.039+0000 7f4b9bfff640 10 monclient(hunting): _renew_subs 2024-09-15T20:49:00.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.039+0000 7f4baad9d640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:00.242 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.243+0000 7f4baad9d640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:00.646 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:00.647+0000 7f4baad9d640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:01.451 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:01.451+0000 7f4baad9d640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 10 monclient(hunting): tick 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 10 monclient(hunting): _check_auth_tickets 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 1 monclient(hunting): continuing hunt 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 10 monclient(hunting): _reopen_session rank -1 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.78:6790/0 conn(0x7f4b8c0036d0 legacy=0x7f4b8c005b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.78:6789/0 conn(0x7f4bac135b60 legacy=0x7f4b8c0093e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:49:03.039 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 10 monclient(hunting): picked mon.noname-b con 0x7f4b8c0106d0 addr v1:172.21.15.78:6790/0 2024-09-15T20:49:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 10 monclient(hunting): picked mon.noname-c con 0x7f4bac13c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:49:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 10 monclient(hunting): picked mon.noname-a con 0x7f4b8c0129c0 addr v1:172.21.15.78:6789/0 2024-09-15T20:49:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b8c011700 con 0x7f4b8c0129c0 2024-09-15T20:49:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac06ecf0 con 0x7f4bac13c220 2024-09-15T20:49:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 1 -- 172.21.15.78:0/3346881759 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b8c009b10 con 0x7f4b8c0106d0 2024-09-15T20:49:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4b9bfff640 10 monclient(hunting): _renew_subs 2024-09-15T20:49:03.040 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.039+0000 7f4ba9d9b640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:03.243 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.243+0000 7f4ba9d9b640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:03.647 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:03.647+0000 7f4ba9d9b640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:04.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 283903840 0 0) 0x7f4b8c009b10 con 0x7f4b8c0106d0 2024-09-15T20:49:04.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient(hunting): my global_id is 154166 2024-09-15T20:49:04.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b94003530 con 0x7f4b8c0106d0 2024-09-15T20:49:04.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1171312879 0 0) 0x7f4b94003530 con 0x7f4b8c0106d0 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.146:6789/0 conn(0x7f4bac13c220 legacy=0x7f4b8c00cbe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.78:6789/0 conn(0x7f4b8c0129c0 legacy=0x7f4b8c014db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: _finish_hunting 0 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 monclient: found mon.noname-b 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b8c009d80 con 0x7f4b8c0106d0 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: _finish_auth 0 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: _check_auth_tickets 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b9c001620 con 0x7f4b8c0106d0 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 5 monclient: authenticate success, global_id 154166 2024-09-15T20:49:04.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: dump: 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b9c0017c0 con 0x7f4b8c0106d0 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: handle_config config(0 keys) 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 1 -- 172.21.15.78:0/3346881759 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b9c001c10 con 0x7f4b8c0106d0 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 10 monclient: get_monmap_and_config success 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 4 set_mon_vals no callback set 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4ba959a640 10 monclient: dump: 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.279 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.280 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.280 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.280 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.280 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.280 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 10 monclient: shutdown 2024-09-15T20:49:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:49:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 1 -- 172.21.15.78:0/3346881759 >> v1:172.21.15.78:6790/0 conn(0x7f4b8c0106d0 legacy=0x7f4b8c010b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 1 -- 172.21.15.78:0/3346881759 shutdown_connections 2024-09-15T20:49:04.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 1 -- 172.21.15.78:0/3346881759 >> 172.21.15.78:0/3346881759 conn(0x7f4bac0a63c0 msgr2=0x7f4bac0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 1 -- 172.21.15.78:0/3346881759 shutdown_connections 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.275+0000 7f4bb0b67640 1 -- 172.21.15.78:0/3346881759 wait complete. 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: build_initial_monmap 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: monmap: 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:49:04.285 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 librados: starting msgr at 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 librados: starting objecter 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 Processor -- start 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 -- start start 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 librados: setting wanted keys 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 librados: calling monclient init 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: init 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: _reopen_session rank -1 2024-09-15T20:49:04.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient(hunting): picked mon.noname-c con 0x7f4b8c0106d0 addr v1:172.21.15.78:6790/0 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient(hunting): picked mon.noname-b con 0x7f4b8c0129c0 addr v1:172.21.15.146:6789/0 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient(hunting): picked mon.noname-a con 0x7f4bac13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac13b060 con 0x7f4bac13c220 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac09d180 con 0x7f4b8c0129c0 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4bac09e310 con 0x7f4b8c0106d0 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient(hunting): _renew_subs 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient(hunting): authenticate will time out at 2205.165658s 2024-09-15T20:49:04.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4baa59c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f4b8c0106d0 0x7f4bac132cc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:55446/0 (socket says 172.21.15.78:55446) 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4baa59c640 1 -- 172.21.15.78:0/1269288653 learned_addr learned my addr 172.21.15.78:0/1269288653 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4ba9d9b640 1 -- 172.21.15.78:0/1269288653 >> v1:172.21.15.146:6789/0 conn(0x7f4b8c0129c0 legacy=0x7f4bac1d1ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 client.?.objecter ms_handle_connect 0x7f4bac13c220 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 client.?.objecter ms_handle_connect 0x7f4b8c0106d0 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1577800208 0 0) 0x7f4bac09e310 con 0x7f4b8c0106d0 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient(hunting): my global_id is 154169 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4b8c0112e0 con 0x7f4b8c0106d0 2024-09-15T20:49:04.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1821428515 0 0) 0x7f4b8c0112e0 con 0x7f4b8c0106d0 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f4bac09e310 con 0x7f4b8c0106d0 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b9c001bd0 con 0x7f4b8c0106d0 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient(hunting): dump: 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.289 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 226183604 0 0) 0x7f4bac09e310 con 0x7f4b8c0106d0 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 >> v1:172.21.15.146:6789/0 conn(0x7f4b8c0129c0 legacy=0x7f4bac1d1ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 >> v1:172.21.15.78:6789/0 conn(0x7f4bac13c220 legacy=0x7f4bac1d53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient: _finish_hunting 0 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 monclient: found mon.c 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bac13b500 con 0x7f4b8c0106d0 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient: _finish_auth 0 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient: _check_auth_tickets 2024-09-15T20:49:04.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:49:04.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 5 monclient: authenticate success, global_id 154169 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: _renew_subs 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f4bac09f330 con 0x7f4b8c0106d0 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 client.154169.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f4b9c0038b0 con 0x7f4b8c0106d0 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: _renew_subs 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f4bac09db30 con 0x7f4b8c0106d0 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 1 librados: init done 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f4b9c003c40 con 0x7f4b8c0106d0 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.292 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 10 monclient: dump: 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4ba8d99640 4 set_mon_vals no callback set 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f4b9c003de0 con 0x7f4b8c0106d0 2024-09-15T20:49:04.293 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.279+0000 7f4bb0b67640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4bb0b67640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4bb0b67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bac135070 con 0x7f4b8c0106d0 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7f4b9c04b6c0 con 0x7f4b8c0106d0 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4b9affd640 10 client.154169.objecter ms_dispatch 0x7f4bac000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4b9affd640 3 client.154169.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4b9affd640 3 client.154169.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4b9affd640 20 client.154169.objecter dump_active .. 0 homeless 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4ba9d9b640 10 monclient: get_auth_request con 0x7f4b8c03f630 auth_method 0 2024-09-15T20:49:04.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.283+0000 7f4b9affd640 10 client.154169.objecter ms_handle_connect 0x7f4b8c03f630 2024-09-15T20:49:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.287+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f4b9c016970 con 0x7f4b8c0106d0 2024-09-15T20:49:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.287+0000 7f4b9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:49:04.295 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.287+0000 7f4b9affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:49:04.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:49:04.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:49:04.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:49:04.514 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4bac06ff40 con 0x7f4b8c0106d0 2024-09-15T20:49:04.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4b9affd640 1 -- 172.21.15.78:0/1269288653 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1400 (unknown 254746657 0 2963183756) 0x7f4b9c016270 con 0x7f4b8c0106d0 2024-09-15T20:49:04.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4b9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:49:04.515 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4b9affd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:49:04.515 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":74,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 10 librados: watch_flush enter 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 10 librados: watch_flush exit 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 20 client.154169.objecter shutdown clearing up homeless session... 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 10 client.154169.objecter successfully canceled tick 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 >> v1:172.21.15.146:6812/435598642 conn(0x7f4b8c03f630 legacy=0x7f4b8c041af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 10 monclient: shutdown 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 >> v1:172.21.15.78:6790/0 conn(0x7f4b8c0106d0 legacy=0x7f4bac132cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 shutdown_connections 2024-09-15T20:49:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 >> 172.21.15.78:0/1269288653 conn(0x7f4bac0a63c0 msgr2=0x7f4bac072ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:49:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 shutdown_connections 2024-09-15T20:49:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 1 -- 172.21.15.78:0/1269288653 wait complete. 2024-09-15T20:49:04.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.515+0000 7f4bb0b67640 1 librados: shutdown 2024-09-15T20:49:04.532 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-09-15T20:49:04.532 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-09-15T20:49:04.532 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.651+0000 7f9544067640 10 monclient: get_monmap_and_config 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.651+0000 7f9544067640 10 monclient: build_initial_monmap 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.651+0000 7f9544067640 10 monclient: monmap: 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:49:04.655453+0000 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:49:04.655453+0000 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:49:04.653 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:49:04.654 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.651+0000 7f9544067640 1 Processor -- start 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.651+0000 7f9544067640 1 -- start start 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.651+0000 7f9544067640 10 monclient: init 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: _reopen_session rank -1 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): picked mon.noname-b con 0x7f953c074630 addr v1:172.21.15.78:6790/0 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): picked mon.noname-c con 0x7f953c09d950 addr v1:172.21.15.146:6789/0 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): picked mon.noname-a con 0x7f953c074a10 addr v1:172.21.15.78:6789/0 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f953c134bc0 con 0x7f953c074a10 2024-09-15T20:49:04.657 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f953c09f900 con 0x7f953c09d950 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f953c09fbe0 con 0x7f953c074630 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): _renew_subs 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): authenticate will time out at 2205.540415s 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9541602640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7f953c09d950 legacy=0x7f953c1344b0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9541e03640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f953c074630 0x7f953c133da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:55460/0 (socket says 172.21.15.78:55460) 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9541e03640 1 -- 172.21.15.78:0/783661556 learned_addr learned my addr 172.21.15.78:0/783661556 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2612532325 0 0) 0x7f953c09fbe0 con 0x7f953c074630 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient(hunting): my global_id is 154172 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9528003530 con 0x7f953c074630 2024-09-15T20:49:04.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018772893 0 0) 0x7f953c134bc0 con 0x7f953c074a10 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient(hunting): my global_id is 144255 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f953c09fbe0 con 0x7f953c074a10 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 954780005 0 0) 0x7f953c09fbe0 con 0x7f953c074a10 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 >> v1:172.21.15.78:6790/0 conn(0x7f953c074630 legacy=0x7f953c133da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 >> v1:172.21.15.146:6789/0 conn(0x7f953c09d950 legacy=0x7f953c1344b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: _finish_hunting 0 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 monclient: found mon.noname-a 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f953c09fec0 con 0x7f953c074a10 2024-09-15T20:49:04.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: _finish_auth 0 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: _check_auth_tickets 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9534001a00 con 0x7f953c074a10 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 5 monclient: authenticate success, global_id 144255 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: dump: 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.660 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9534001c20 con 0x7f953c074a10 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: handle_config config(0 keys) 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 1 -- 172.21.15.78:0/783661556 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9534002030 con 0x7f953c074a10 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: get_monmap_and_config success 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 4 set_mon_vals no callback set 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9540e01640 10 monclient: dump: 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: shutdown 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- 172.21.15.78:0/783661556 >> v1:172.21.15.78:6789/0 conn(0x7f953c074a10 legacy=0x7f953c13d150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- 172.21.15.78:0/783661556 shutdown_connections 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- 172.21.15.78:0/783661556 >> 172.21.15.78:0/783661556 conn(0x7f953c0a63c0 msgr2=0x7f953c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- 172.21.15.78:0/783661556 shutdown_connections 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- 172.21.15.78:0/783661556 wait complete. 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: build_initial_monmap 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: monmap: 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:49:04.663 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:49:04.664 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 librados: starting msgr at 2024-09-15T20:49:04.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 librados: starting objecter 2024-09-15T20:49:04.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 Processor -- start 2024-09-15T20:49:04.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- start start 2024-09-15T20:49:04.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 librados: setting wanted keys 2024-09-15T20:49:04.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 librados: calling monclient init 2024-09-15T20:49:04.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: init 2024-09-15T20:49:04.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: _reopen_session rank -1 2024-09-15T20:49:04.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:49:04.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): picked mon.noname-a con 0x7f953c074630 addr v1:172.21.15.78:6789/0 2024-09-15T20:49:04.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): picked mon.noname-c con 0x7f953c074a10 addr v1:172.21.15.78:6790/0 2024-09-15T20:49:04.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): picked mon.noname-b con 0x7f953c09d950 addr v1:172.21.15.146:6789/0 2024-09-15T20:49:04.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f953c070b00 con 0x7f953c074630 2024-09-15T20:49:04.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f953c12e730 con 0x7f953c09d950 2024-09-15T20:49:04.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9541e03640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f953c074630 0x7f953c0735e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34154/0 (socket says 172.21.15.78:34154) 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f953c0728a0 con 0x7f953c074a10 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9541e03640 1 -- 172.21.15.78:0/1530711696 learned_addr learned my addr 172.21.15.78:0/1530711696 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): _renew_subs 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.655+0000 7f9544067640 10 monclient(hunting): authenticate will time out at 2205.543930s 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 client.?.objecter ms_handle_connect 0x7f953c074630 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9542604640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.146:6789/0 conn(0x7f953c09d950 legacy=0x7f953c12e020 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1976548507 0 0) 0x7f953c070b00 con 0x7f953c074630 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): my global_id is 144258 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9520004ec0 con 0x7f953c074630 2024-09-15T20:49:04.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 client.?.objecter ms_handle_connect 0x7f953c074a10 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 636587690 0 0) 0x7f953c0728a0 con 0x7f953c074a10 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): my global_id is 154175 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f953c070b00 con 0x7f953c074a10 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3145799612 0 0) 0x7f9520004ec0 con 0x7f953c074630 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f953c0728a0 con 0x7f953c074630 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f95380013a0 con 0x7f953c074630 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:49:04.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): dump: 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 751670634 0 0) 0x7f953c070b00 con 0x7f953c074a10 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9520004ec0 con 0x7f953c074a10 2024-09-15T20:49:04.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f952c004190 con 0x7f953c074a10 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): dump: 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4241660544 0 0) 0x7f953c0728a0 con 0x7f953c074630 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.78:6790/0 conn(0x7f953c074a10 legacy=0x7f953c12d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.146:6789/0 conn(0x7f953c09d950 legacy=0x7f953c12e020 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: _finish_hunting 0 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 monclient: found mon.a 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f953c0725c0 con 0x7f953c074630 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: _finish_auth 0 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: _check_auth_tickets 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 5 monclient: authenticate success, global_id 144258 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 10 monclient: _renew_subs 2024-09-15T20:49:04.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f953c071660 con 0x7f953c074630 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 10 client.144258.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 10 monclient: _renew_subs 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f953c1d17d0 con 0x7f953c074630 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 1 librados: init done 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f9538001600 con 0x7f953c074630 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f9538004c40 con 0x7f953c074630 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9533fff640 4 set_mon_vals no callback set 2024-09-15T20:49:04.676 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: dump: 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f95240065d0 con 0x7f9524002870 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 client.144258.objecter ms_handle_connect 0x7f9524002870 2024-09-15T20:49:04.677 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 client.144258.objecter resend_mon_ops 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f9538010cb0 con 0x7f953c074630 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7f953804b7d0 con 0x7f953c074630 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 client.144258.objecter ms_dispatch 0x7f953c000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 3 client.144258.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 3 client.144258.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 20 client.144258.objecter dump_active .. 0 homeless 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1776636802 0 0) 0x7f95240065d0 con 0x7f9524002870 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient(hunting): my global_id is 144261 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f953c0728a0 con 0x7f9524002870 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3713863502 0 0) 0x7f953c0728a0 con 0x7f9524002870 2024-09-15T20:49:04.678 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f95240065d0 con 0x7f9524002870 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f9542604640 10 monclient: get_auth_request con 0x7f952003df90 auth_method 0 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f952c004810 con 0x7f9524002870 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: dump: 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.679 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1030945540 0 0) 0x7f95240065d0 con 0x7f9524002870 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f952400a170 con 0x7f9524002870 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 client.144258.objecter ms_handle_connect 0x7f952003df90 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7f952c006b30 con 0x7f9524002870 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.78:6789/0 conn(0x7f9524002870 legacy=0x7f9524004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.659+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.78:6789/0 conn(0x7f9524002870 legacy=0x7f9524004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7f9524009e60 con 0x7f9524009120 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 client.144258.objecter ms_handle_connect 0x7f9524009120 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 client.144258.objecter resend_mon_ops 2024-09-15T20:49:04.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2924354373 0 0) 0x7f9524009e60 con 0x7f9524009120 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient(hunting): my global_id is 144264 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f95240065d0 con 0x7f9524009120 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 2258496712 0 0) 0x7f95240065d0 con 0x7f9524009120 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f9524009e60 con 0x7f9524009120 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f952c004ec0 con 0x7f9524009120 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient: dump: 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.681 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3694488679 0 0) 0x7f9524009e60 con 0x7f9524009120 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f952400c400 con 0x7f9524009120 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4296 (unknown 0 0 3348299620) 0x7f952c0063f0 con 0x7f9524009120 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:49:04.682 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.78:6789/0 conn(0x7f9524009120 legacy=0x7f95240095b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f95327fc640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.78:6789/0 conn(0x7f9524009120 legacy=0x7f95240095b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 74, 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 146562, 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:49:04.683 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:49:04.684 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:49:04.685 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:49:04.686 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:49:04.687 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:49:04.688 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:49:04.689 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.690 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:49:04.691 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:49:04.692 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:49:04.693 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 7 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:49:04.694 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:49:04.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 10 librados: watch_flush enter 2024-09-15T20:49:04.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 10 librados: watch_flush exit 2024-09-15T20:49:04.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 20 client.144258.objecter shutdown clearing up homeless session... 2024-09-15T20:49:04.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 10 client.144258.objecter successfully canceled tick 2024-09-15T20:49:04.696 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.146:6812/435598642 conn(0x7f952003df90 legacy=0x7f9520040450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 10 monclient: shutdown 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 >> v1:172.21.15.78:6789/0 conn(0x7f953c074630 legacy=0x7f953c0735e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9542604640 1 -- 172.21.15.78:0/1530711696 reap_dead start 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9542604640 1 -- 172.21.15.78:0/1530711696 reap_dead start 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 shutdown_connections 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 >> 172.21.15.78:0/1530711696 conn(0x7f953c0a63c0 msgr2=0x7f953c133980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 shutdown_connections 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 1 -- 172.21.15.78:0/1530711696 wait complete. 2024-09-15T20:49:04.697 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.667+0000 7f9544067640 1 librados: shutdown 2024-09-15T20:49:04.697 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:49:04.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-09-15T20:49:04.700 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:49:04.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: get_monmap_and_config 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: build_initial_monmap 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: monmap: 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:49:04.817135+0000 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:49:04.817135+0000 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:49:04.830 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 Processor -- start 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- start start 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: init 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: _reopen_session rank -1 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf8135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf80a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:49:04.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf80745e0 con 0x7fbcf80a24e0 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf806ecf0 con 0x7fbcf8135b60 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf806efd0 con 0x7fbcf813c220 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient(hunting): _renew_subs 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient(hunting): authenticate will time out at 2205.701732s 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf7fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fbcf813c220 0x7fbcf813e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:55480/0 (socket says 172.21.15.78:55480) 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf77fe640 1 -- >> v1:172.21.15.146:6789/0 conn(0x7fbcf8135b60 legacy=0x7fbcf8135f40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf7fff640 1 -- 172.21.15.78:0/3767028986 learned_addr learned my addr 172.21.15.78:0/3767028986 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbcf80a24e0 0x7fbcf813ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34172/0 (socket says 172.21.15.78:34172) 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3912151243 0 0) 0x7fbcf80745e0 con 0x7fbcf80a24e0 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient(hunting): my global_id is 144267 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbce4003530 con 0x7fbcf80a24e0 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457429449 0 0) 0x7fbcf806efd0 con 0x7fbcf813c220 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient(hunting): my global_id is 154178 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcf80745e0 con 0x7fbcf813c220 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3396087245 0 0) 0x7fbce4003530 con 0x7fbcf80a24e0 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 >> v1:172.21.15.78:6790/0 conn(0x7fbcf813c220 legacy=0x7fbcf813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 >> v1:172.21.15.146:6789/0 conn(0x7fbcf8135b60 legacy=0x7fbcf8135f40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-09-15T20:49:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: _finish_hunting 0 2024-09-15T20:49:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 monclient: found mon.noname-a 2024-09-15T20:49:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:49:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:49:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf8136650 con 0x7fbcf80a24e0 2024-09-15T20:49:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: _finish_auth 0 2024-09-15T20:49:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: _check_auth_tickets 2024-09-15T20:49:04.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbce8001a20 con 0x7fbcf80a24e0 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: dump: 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.835 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.836 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 5 monclient: authenticate success, global_id 144267 2024-09-15T20:49:04.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:49:04.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbce8001c40 con 0x7fbcf80a24e0 2024-09-15T20:49:04.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:49:04.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: get_monmap_and_config success 2024-09-15T20:49:04.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 4 set_mon_vals no callback set 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 1 -- 172.21.15.78:0/3767028986 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbce8002050 con 0x7fbcf80a24e0 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcf67fc640 10 monclient: dump: 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.837 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.838 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: shutdown 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- 172.21.15.78:0/3767028986 >> v1:172.21.15.78:6789/0 conn(0x7fbcf80a24e0 legacy=0x7fbcf813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- 172.21.15.78:0/3767028986 shutdown_connections 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- 172.21.15.78:0/3767028986 >> 172.21.15.78:0/3767028986 conn(0x7fbcf80a63c0 msgr2=0x7fbcf80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- 172.21.15.78:0/3767028986 shutdown_connections 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 -- 172.21.15.78:0/3767028986 wait complete. 2024-09-15T20:49:04.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: build_initial_monmap 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 10 monclient: monmap: 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 librados: starting msgr at 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.815+0000 7fbcfe18d640 1 librados: starting objecter 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 Processor -- start 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 -- start start 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 librados: setting wanted keys 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 librados: calling monclient init 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient: init 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient: _reopen_session rank -1 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient(hunting): picked mon.noname-b con 0x7fbcf80a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient(hunting): picked mon.noname-a con 0x7fbcf8135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient(hunting): picked mon.noname-c con 0x7fbcf813c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf809e2b0 con 0x7fbcf8135b60 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf813bbb0 con 0x7fbcf80a24e0 2024-09-15T20:49:04.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbcf809d180 con 0x7fbcf813c220 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient(hunting): _renew_subs 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient(hunting): authenticate will time out at 2205.704630s 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcf6ffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fbcf8135b60 0x7fbcf809dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:34188/0 (socket says 172.21.15.78:34188) 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcf6ffd640 1 -- 172.21.15.78:0/1277482529 learned_addr learned my addr 172.21.15.78:0/1277482529 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcf77fe640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.146:6789/0 conn(0x7fbcf80a24e0 legacy=0x7fbcf809d490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v1:172.21.15.146:6789/0 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 client.?.objecter ms_handle_connect 0x7fbcf8135b60 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 client.?.objecter ms_handle_connect 0x7fbcf813c220 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2338364291 0 0) 0x7fbcf809e2b0 con 0x7fbcf8135b60 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): my global_id is 144270 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcd8004ef0 con 0x7fbcf8135b60 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 788385322 0 0) 0x7fbcf809d180 con 0x7fbcf813c220 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): my global_id is 154181 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcf809e2b0 con 0x7fbcf813c220 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2204335912 0 0) 0x7fbcd8004ef0 con 0x7fbcf8135b60 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcf809d180 con 0x7fbcf8135b60 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbce8002050 con 0x7fbcf8135b60 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:49:04.843 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): dump: 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 571134372 0 0) 0x7fbcf809e2b0 con 0x7fbcf813c220 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcd8004ef0 con 0x7fbcf813c220 2024-09-15T20:49:04.844 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbcf0001220 con 0x7fbcf813c220 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): dump: 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.845 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 49878495 0 0) 0x7fbcf809d180 con 0x7fbcf8135b60 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.78:6790/0 conn(0x7fbcf813c220 legacy=0x7fbcf81d4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.146:6789/0 conn(0x7fbcf80a24e0 legacy=0x7fbcf809d490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: _finish_hunting 0 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 monclient: found mon.a 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcf8130a40 con 0x7fbcf8135b60 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: _finish_auth 0 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: _check_auth_tickets 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 5 monclient: authenticate success, global_id 144270 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient: _renew_subs 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:49:04.846 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fbcf812fbd0 con 0x7fbcf8135b60 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 client.144270.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient: _renew_subs 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fbcf813aeb0 con 0x7fbcf8135b60 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 librados: init done 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fbce8001540 con 0x7fbcf8135b60 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbce8005060 con 0x7fbcf8135b60 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcf5ffb640 4 set_mon_vals no callback set 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: dump: 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.847 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbcdc0065d0 con 0x7fbcdc002870 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 client.144270.objecter ms_handle_connect 0x7fbcdc002870 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 client.144270.objecter resend_mon_ops 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2783684614 0 0) 0x7fbcdc0065d0 con 0x7fbcdc002870 2024-09-15T20:49:04.848 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient(hunting): my global_id is 154184 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcf809d180 con 0x7fbcdc002870 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fbce8005280 con 0x7fbcf8135b60 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(690..690 src has 379..690) ==== 5532+0+0 (unknown 3605475098 0 0) 0x7fbce804bda0 con 0x7fbcf8135b60 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 client.144270.objecter ms_dispatch 0x7fbcf8000ba0 osd_map(690..690 src has 379..690) 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 3 client.144270.objecter handle_osd_map got epochs [690,690] > 0 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 3 client.144270.objecter handle_osd_map decoding full epoch 690 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 20 client.144270.objecter dump_active .. 0 homeless 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 650624097 0 0) 0x7fbcf809d180 con 0x7fbcdc002870 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcdc0065d0 con 0x7fbcdc002870 2024-09-15T20:49:04.849 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbce0003eb0 con 0x7fbcdc002870 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: dump: 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.850 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3718140984 0 0) 0x7fbcdc0065d0 con 0x7fbcdc002870 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbcdc00a170 con 0x7fbcdc002870 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbcf7fff640 10 monclient: get_auth_request con 0x7fbcd803e070 auth_method 0 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fbce00054d0 con 0x7fbcdc002870 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.78:6790/0 conn(0x7fbcdc002870 legacy=0x7fbcdc004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.78:6790/0 conn(0x7fbcdc002870 legacy=0x7fbcdc004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.819+0000 7fbceffff640 10 client.144270.objecter ms_handle_connect 0x7fbcd803e070 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fbcdc009e80 con 0x7fbcdc009140 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 client.144270.objecter ms_handle_connect 0x7fbcdc009140 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 client.144270.objecter resend_mon_ops 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 649160452 0 0) 0x7fbcdc009e80 con 0x7fbcdc009140 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient(hunting): _negotiate 2024-09-15T20:49:04.851 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient(hunting): my global_id is 154187 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbcdc0065d0 con 0x7fbcdc009140 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1212816814 0 0) 0x7fbcdc0065d0 con 0x7fbcdc009140 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fbcdc009e80 con 0x7fbcdc009140 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fbce00042f0 con 0x7fbcdc009140 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient: dump: 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.852 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 261756716 0 0) 0x7fbcdc009e80 con 0x7fbcdc009140 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbcdc00c420 con 0x7fbcdc009140 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3959 (unknown 0 0 940947418) 0x7fbce0005700 con 0x7fbcdc009140 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.78:6790/0 conn(0x7fbcdc009140 legacy=0x7fbcdc0095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.853 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbceffff640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.78:6790/0 conn(0x7fbcdc009140 legacy=0x7fbcdc0095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 74, 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 32811, 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:49:04.854 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:49:04.855 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:49:04.856 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:49:04.857 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:49:04.858 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:49:04.859 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.860 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:49:04.861 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.862 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:49:04.863 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:49:04.864 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:49:04.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 10 librados: watch_flush enter 2024-09-15T20:49:04.866 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 10 librados: watch_flush exit 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 20 client.144270.objecter shutdown clearing up homeless session... 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 10 client.144270.objecter successfully canceled tick 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.146:6812/435598642 conn(0x7fbcd803e070 legacy=0x7fbcd8040530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 10 monclient: shutdown 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 >> v1:172.21.15.78:6789/0 conn(0x7fbcf8135b60 legacy=0x7fbcf809dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcf7fff640 1 -- 172.21.15.78:0/1277482529 reap_dead start 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcf7fff640 1 -- 172.21.15.78:0/1277482529 reap_dead start 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 shutdown_connections 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 >> 172.21.15.78:0/1277482529 conn(0x7fbcf80a63c0 msgr2=0x7fbcf812e770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 shutdown_connections 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 1 -- 172.21.15.78:0/1277482529 wait complete. 2024-09-15T20:49:04.867 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:49:04.827+0000 7fbcfe18d640 1 librados: shutdown 2024-09-15T20:49:04.868 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-09-15T20:49:04.868 INFO:tasks.mon_thrash.mon_thrasher:waiting for 90.0 secs before reviving monitors 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlush (7915 ms) 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FailedFlush 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FailedFlush (13094 ms) 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.Flush 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.Flush (7925 ms) 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushSnap 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushSnap (13018 ms) 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.FlushTryFlushRaces 2024-09-15T20:49:16.195 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.FlushTryFlushRaces (8032 ms) 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TryFlushReadRace 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TryFlushReadRace (27892 ms) 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetRead 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: hmm, no HitSet yet 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: ok, hit_set contains 332:602f83fe:::foo:head 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetRead (9094 ms) 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.HitSetTrim 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,0 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: first is 1726433242 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,0 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,0 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,0 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,0 2024-09-15T20:49:16.196 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,1726433247,1726433248,0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,1726433247,1726433248,0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,1726433247,1726433248,0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,1726433247,1726433248,1726433250,1726433251,0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,1726433247,1726433248,1726433250,1726433251,0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433242,1726433244,1726433245,1726433247,1726433248,1726433250,1726433251,0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: got ls 1726433245,1726433247,1726433248,1726433250,1726433251,1726433253,1726433254,0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: first now 1726433245, trimmed 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.HitSetTrim (20959 ms) 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.PromoteOn2ndRead 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: foo0 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: verifying foo0 is eventually promoted 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.PromoteOn2ndRead (14000 ms) 2024-09-15T20:49:16.197 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ProxyRead 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ProxyRead (18109 ms) 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.CachePin 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.CachePin (22004 ms) 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetRedirectRead 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetRedirectRead (19960 ms) 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.SetChunkRead 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.SetChunkRead (18042 ms) 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.ManifestPromoteRead 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.ManifestPromoteRead (3016 ms) 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ RUN ] LibRadosTwoPoolsECPP.TrySetDedupTier 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ OK ] LibRadosTwoPoolsECPP.TrySetDedupTier (2851 ms) 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] 22 tests from LibRadosTwoPoolsECPP (288927 ms total) 2024-09-15T20:49:16.198 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: 2024-09-15T20:49:16.199 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [----------] Global test environment tear-down 2024-09-15T20:49:16.199 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [==========] 77 tests from 4 test suites ran. (987523 ms total) 2024-09-15T20:49:16.199 INFO:tasks.workunit.client.0.smithi146.stdout: api_tier_pp: [ PASSED ] 77 tests. 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28196 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28196 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28358 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28358 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28753 2024-09-15T20:49:16.200 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28753 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28599 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28599 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28427 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28427 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28228 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28228 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28495 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28495 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.201 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28865 2024-09-15T20:49:16.202 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28865 2024-09-15T20:49:16.202 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.202 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28928 2024-09-15T20:49:16.202 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28928 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28266 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28266 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28768 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28768 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28180 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28180 2024-09-15T20:49:16.207 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28212 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28212 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28674 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28674 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28172 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28172 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28188 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28188 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.208 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29039 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 29039 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28407 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28407 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29093 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 29093 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28457 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28457 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28799 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28799 2024-09-15T20:49:16.209 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29152 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 29152 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28416 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28416 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28721 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28721 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28995 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28995 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28780 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28780 2024-09-15T20:49:16.210 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29059 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 29059 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28647 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28647 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28333 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28333 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29124 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 29124 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28609 2024-09-15T20:49:16.211 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28609 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28164 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28164 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28204 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28204 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28577 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28577 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28220 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28220 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.212 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28626 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28626 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28639 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28639 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=28964 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 28964 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ for t in "${!pids[@]}" 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ pid=29169 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ wait 29169 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ exit 0 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ cleanup 2024-09-15T20:49:16.213 INFO:tasks.workunit.client.0.smithi146.stderr:+ pkill -P 28158 2024-09-15T20:49:16.218 INFO:tasks.workunit.client.0.smithi146.stderr:+ true 2024-09-15T20:49:16.219 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-15T20:49:16.219 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-09-15T20:49:16.233 INFO:tasks.workunit:Stopping ['rados/test.sh unit_test_scan'] on client.0... 2024-09-15T20:49:16.234 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-09-15T20:49:16.597 DEBUG:teuthology.parallel:result is None 2024-09-15T20:49:16.598 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T20:49:16.610 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T20:49:16.610 DEBUG:teuthology.orchestra.run.smithi146:> rmdir -- /home/ubuntu/cephtest/mnt.0 2024-09-15T20:49:16.665 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2024-09-15T20:49:16.665 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-09-15T20:49:16.674 INFO:tasks.mon_thrash:joining mon_thrasher 2024-09-15T20:50:34.869 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-09-15T20:50:34.869 INFO:tasks.ceph.mon.b:Restarting daemon 2024-09-15T20:50:34.870 DEBUG:teuthology.orchestra.run.smithi146:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-09-15T20:50:34.873 INFO:tasks.ceph.mon.b:Started 2024-09-15T20:50:34.873 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:50:34.873 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient: get_monmap_and_config 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient: monmap: 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:35.006128+0000 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:35.006128+0000 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.004 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:35.005 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:35.005 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 1 Processor -- start 2024-09-15T20:50:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 1 -- start start 2024-09-15T20:50:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient: init 2024-09-15T20:50:35.011 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient(hunting): picked mon.noname-c con 0x7f0eb4135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient(hunting): picked mon.noname-b con 0x7f0eb40a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient(hunting): picked mon.noname-a con 0x7f0eb413c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb40745e0 con 0x7f0eb413c220 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb406ecf0 con 0x7f0eb4135b60 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb406efd0 con 0x7f0eb40a24e0 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebc56f640 10 monclient(hunting): authenticate will time out at 2295.891285s 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebab0c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0eb413c220 0x7f0eb413e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46044/0 (socket says 172.21.15.78:46044) 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0ebab0c640 1 -- 172.21.15.78:0/2605252627 learned_addr learned my addr 172.21.15.78:0/2605252627 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1277022991 0 0) 0x7f0eb40745e0 con 0x7f0eb413c220 2024-09-15T20:50:35.012 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 10 monclient(hunting): my global_id is 144273 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0ea0003530 con 0x7f0eb413c220 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2670650010 0 0) 0x7f0eb406efd0 con 0x7f0eb40a24e0 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 10 monclient(hunting): my global_id is 154190 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eb40745e0 con 0x7f0eb40a24e0 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3261504530 0 0) 0x7f0eb40745e0 con 0x7f0eb40a24e0 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.001+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 >> v1:172.21.15.146:6789/0 conn(0x7f0eb4135b60 legacy=0x7f0eb4135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.013 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 >> v1:172.21.15.78:6789/0 conn(0x7f0eb413c220 legacy=0x7f0eb413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 1 monclient: found mon.noname-b 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb4136650 con 0x7f0eb40a24e0 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0ea4001680 con 0x7f0eb40a24e0 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 5 monclient: authenticate success, global_id 154190 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:35.014 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: dump: 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ea4001820 con 0x7f0eb40a24e0 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.015 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 1 -- 172.21.15.78:0/2605252627 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0ea4001c30 con 0x7f0eb40a24e0 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 4 set_mon_vals no callback set 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb9309640 10 monclient: dump: 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.016 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: shutdown 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2605252627 >> v1:172.21.15.78:6790/0 conn(0x7f0eb40a24e0 legacy=0x7f0eb413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2605252627 shutdown_connections 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2605252627 >> 172.21.15.78:0/2605252627 conn(0x7f0eb40a63c0 msgr2=0x7f0eb40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2605252627 shutdown_connections 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2605252627 wait complete. 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: monmap: 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.017 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 librados: starting msgr at 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 librados: starting objecter 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 Processor -- start 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- start start 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 librados: setting wanted keys 2024-09-15T20:50:35.018 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 librados: calling monclient init 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: init 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient(hunting): picked mon.noname-a con 0x7f0eb40a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient(hunting): picked mon.noname-c con 0x7f0eb4135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient(hunting): picked mon.noname-b con 0x7f0eb413c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb4138080 con 0x7f0eb40a24e0 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb413bbb0 con 0x7f0eb413c220 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0eb413a690 con 0x7f0eb4135b60 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient(hunting): authenticate will time out at 2295.894269s 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eba30b640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0eb40a24e0 0x7f0eb4136bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46056/0 (socket says 172.21.15.78:46056) 2024-09-15T20:50:35.019 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eba30b640 1 -- 172.21.15.78:0/2914083255 learned_addr learned my addr 172.21.15.78:0/2914083255 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 client.?.objecter ms_handle_connect 0x7f0eb40a24e0 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 client.?.objecter ms_handle_connect 0x7f0eb4135b60 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 349570855 0 0) 0x7f0eb413a690 con 0x7f0eb4135b60 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient(hunting): my global_id is 154193 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e98004ec0 con 0x7f0eb4135b60 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 client.?.objecter ms_handle_connect 0x7f0eb413c220 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1478047667 0 0) 0x7f0e98004ec0 con 0x7f0eb4135b60 2024-09-15T20:50:35.020 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0eb413a690 con 0x7f0eb4135b60 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0ea4001580 con 0x7f0eb4135b60 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient(hunting): dump: 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.021 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2231629396 0 0) 0x7f0eb413a690 con 0x7f0eb4135b60 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 >> v1:172.21.15.146:6789/0 conn(0x7f0eb413c220 legacy=0x7f0eb41d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 >> v1:172.21.15.78:6789/0 conn(0x7f0eb40a24e0 legacy=0x7f0eb4136bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 monclient: found mon.c 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eb41379d0 con 0x7f0eb4135b60 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 5 monclient: authenticate success, global_id 154193 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: _renew_subs 2024-09-15T20:50:35.022 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0eb413a970 con 0x7f0eb4135b60 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 client.154193.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: _renew_subs 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0eb413a3b0 con 0x7f0eb4135b60 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0ebc56f640 1 librados: init done 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0ea4001720 con 0x7f0eb4135b60 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0ea4003a50 con 0x7f0eb4135b60 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:35.023 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eaaffd640 10 monclient: dump: 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.024 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.005+0000 7f0eb8b08640 4 set_mon_vals no callback set 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0ebc56f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0ebc56f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0ebc56f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e9c005090 con 0x7f0eb4135b60 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f0ea4003bf0 con 0x7f0eb4135b60 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f0ea404a2c0 con 0x7f0eb4135b60 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0eaaffd640 10 client.154193.objecter ms_dispatch 0x7f0eb4000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0eaaffd640 3 client.154193.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0eaaffd640 3 client.154193.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0eaaffd640 20 client.154193.objecter dump_active .. 0 homeless 2024-09-15T20:50:35.025 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0eba30b640 10 monclient: get_auth_request con 0x7f0e9803e030 auth_method 0 2024-09-15T20:50:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.009+0000 7f0eaaffd640 10 client.154193.objecter ms_handle_connect 0x7f0e9803e030 2024-09-15T20:50:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.013+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f0ea4015860 con 0x7f0eb4135b60 2024-09-15T20:50:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.013+0000 7f0eaaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.026 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.013+0000 7f0eaaffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.261+0000 7f0ebc56f640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:50:35.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.261+0000 7f0ebc56f640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:50:35.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.261+0000 7f0ebc56f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.261+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0e9c005380 con 0x7f0eb4135b60 2024-09-15T20:50:35.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.261+0000 7f0eaaffd640 1 -- 172.21.15.78:0/2914083255 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 2692922119) 0x7f0ea4015160 con 0x7f0eb4135b60 2024-09-15T20:50:35.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.261+0000 7f0eaaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:50:35.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.261+0000 7f0eaaffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:50:35.265 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:50:35.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 10 librados: watch_flush enter 2024-09-15T20:50:35.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 10 librados: watch_flush exit 2024-09-15T20:50:35.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 20 client.154193.objecter shutdown clearing up homeless session... 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 10 client.154193.objecter successfully canceled tick 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 >> v1:172.21.15.146:6812/435598642 conn(0x7f0e9803e030 legacy=0x7f0e980404f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 10 monclient: shutdown 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 >> v1:172.21.15.78:6790/0 conn(0x7f0eb4135b60 legacy=0x7f0eb41372c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 shutdown_connections 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 >> 172.21.15.78:0/2914083255 conn(0x7f0eb40a63c0 msgr2=0x7f0eb413ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 shutdown_connections 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 1 -- 172.21.15.78:0/2914083255 wait complete. 2024-09-15T20:50:35.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.265+0000 7f0ebc56f640 1 librados: shutdown 2024-09-15T20:50:35.283 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:50:35.283 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-09-15T20:50:35.283 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-09-15T20:50:35.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient: get_monmap_and_config 2024-09-15T20:50:35.396 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient: monmap: 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:35.398616+0000 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:35.398616+0000 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:35.397 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 1 Processor -- start 2024-09-15T20:50:35.400 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 1 -- start start 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient: init 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient(hunting): picked mon.noname-a con 0x7ff3d0135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient(hunting): picked mon.noname-b con 0x7ff3d00a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient(hunting): picked mon.noname-c con 0x7ff3d013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d00745e0 con 0x7ff3d0135b60 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d006ecf0 con 0x7ff3d013c220 2024-09-15T20:50:35.401 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d006efd0 con 0x7ff3d00a24e0 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3d6328640 10 monclient(hunting): authenticate will time out at 2296.283392s 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3c7fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff3d00a24e0 0x7ff3d013ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35258/0 (socket says 172.21.15.78:35258) 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff3d0135b60 0x7ff3d0135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46072/0 (socket says 172.21.15.78:46072) 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3c7fff640 1 -- 172.21.15.78:0/1419402734 learned_addr learned my addr 172.21.15.78:0/1419402734 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3879630489 0 0) 0x7ff3d00745e0 con 0x7ff3d0135b60 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.402 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 10 monclient(hunting): my global_id is 144276 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3bc003530 con 0x7ff3d0135b60 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2215620829 0 0) 0x7ff3d006efd0 con 0x7ff3d00a24e0 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 10 monclient(hunting): my global_id is 154196 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.393+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3d00745e0 con 0x7ff3d00a24e0 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2856414546 0 0) 0x7ff3bc003530 con 0x7ff3d0135b60 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 >> v1:172.21.15.78:6790/0 conn(0x7ff3d00a24e0 legacy=0x7ff3d013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 >> v1:172.21.15.146:6789/0 conn(0x7ff3d013c220 legacy=0x7ff3d013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 1 monclient: found mon.noname-a 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.403 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d0136650 con 0x7ff3d0135b60 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 5 monclient: authenticate success, global_id 144276 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff3c00016a0 con 0x7ff3d0135b60 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: dump: 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.404 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff3c0001840 con 0x7ff3d0135b60 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 1 -- 172.21.15.78:0/1419402734 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff3c0001c50 con 0x7ff3d0135b60 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 4 set_mon_vals no callback set 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cf7fe640 10 monclient: dump: 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.406 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient: shutdown 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- 172.21.15.78:0/1419402734 >> v1:172.21.15.78:6789/0 conn(0x7ff3d0135b60 legacy=0x7ff3d0135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- 172.21.15.78:0/1419402734 shutdown_connections 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- 172.21.15.78:0/1419402734 >> 172.21.15.78:0/1419402734 conn(0x7ff3d00a63c0 msgr2=0x7ff3d00a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- 172.21.15.78:0/1419402734 shutdown_connections 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- 172.21.15.78:0/1419402734 wait complete. 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient: monmap: 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.407 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 librados: starting msgr at 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 librados: starting objecter 2024-09-15T20:50:35.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 Processor -- start 2024-09-15T20:50:35.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- start start 2024-09-15T20:50:35.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 librados: setting wanted keys 2024-09-15T20:50:35.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 librados: calling monclient init 2024-09-15T20:50:35.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient: init 2024-09-15T20:50:35.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:50:35.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient(hunting): picked mon.noname-c con 0x7ff3d00a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient(hunting): picked mon.noname-a con 0x7ff3d0135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient(hunting): picked mon.noname-b con 0x7ff3d013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d01388a0 con 0x7ff3d0135b60 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d009d180 con 0x7ff3d013c220 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3d009ef80 con 0x7ff3d00a24e0 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3d6328640 10 monclient(hunting): authenticate will time out at 2296.286830s 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cffff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff3d00a24e0 0x7ff3d01cceb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35264/0 (socket says 172.21.15.78:35264) 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cffff640 1 -- 172.21.15.78:0/2991392586 learned_addr learned my addr 172.21.15.78:0/2991392586 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 client.?.objecter ms_handle_connect 0x7ff3d0135b60 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 client.?.objecter ms_handle_connect 0x7ff3d00a24e0 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 client.?.objecter ms_handle_connect 0x7ff3d013c220 2024-09-15T20:50:35.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2123812564 0 0) 0x7ff3d01388a0 con 0x7ff3d0135b60 2024-09-15T20:50:35.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 monclient(hunting): my global_id is 144279 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3b0004ec0 con 0x7ff3d0135b60 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3947463342 0 0) 0x7ff3d009ef80 con 0x7ff3d00a24e0 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 10 monclient(hunting): my global_id is 154199 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.397+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3d01388a0 con 0x7ff3d00a24e0 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3937223776 0 0) 0x7ff3b0004ec0 con 0x7ff3d0135b60 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3d009ef80 con 0x7ff3d0135b60 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff3b80012e0 con 0x7ff3d0135b60 2024-09-15T20:50:35.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): dump: 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4027470761 0 0) 0x7ff3d01388a0 con 0x7ff3d00a24e0 2024-09-15T20:50:35.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3b0004ec0 con 0x7ff3d00a24e0 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff3c0001b90 con 0x7ff3d00a24e0 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): dump: 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.420 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3543501386 0 0) 0x7ff3d009ef80 con 0x7ff3d0135b60 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 >> v1:172.21.15.78:6790/0 conn(0x7ff3d00a24e0 legacy=0x7ff3d01cceb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 >> v1:172.21.15.146:6789/0 conn(0x7ff3d013c220 legacy=0x7ff3d01d5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 monclient: found mon.a 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d0139c60 con 0x7ff3d0135b60 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 5 monclient: authenticate success, global_id 144279 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 10 monclient: _renew_subs 2024-09-15T20:50:35.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d0135590 con 0x7ff3d0135b60 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 10 client.144279.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 10 monclient: _renew_subs 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff3d009e4d0 con 0x7ff3d0135b60 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 1 librados: init done 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff3b80047f0 con 0x7ff3d0135b60 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff3b8004b40 con 0x7ff3d0135b60 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3ceffd640 4 set_mon_vals no callback set 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: dump: 2024-09-15T20:50:35.422 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff3b40065d0 con 0x7ff3b4002870 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 client.144279.objecter ms_handle_connect 0x7ff3b4002870 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 client.144279.objecter resend_mon_ops 2024-09-15T20:50:35.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff3b8010bb0 con 0x7ff3d0135b60 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7ff3b804b4b0 con 0x7ff3d0135b60 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 client.144279.objecter ms_dispatch 0x7ff3d0000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 3 client.144279.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 3 client.144279.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 20 client.144279.objecter dump_active .. 0 homeless 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2330098322 0 0) 0x7ff3b40065d0 con 0x7ff3b4002870 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient(hunting): my global_id is 144282 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3d009ef80 con 0x7ff3b4002870 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 4279887556 0 0) 0x7ff3d009ef80 con 0x7ff3b4002870 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3d48c5640 10 monclient: get_auth_request con 0x7ff3b003e000 auth_method 0 2024-09-15T20:50:35.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3b40065d0 con 0x7ff3b4002870 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff3c0004a80 con 0x7ff3b4002870 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: dump: 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.425 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 92614218 0 0) 0x7ff3b40065d0 con 0x7ff3b4002870 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff3b400a170 con 0x7ff3b4002870 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 client.144279.objecter ms_handle_connect 0x7ff3b003e000 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7ff3c0004a80 con 0x7ff3b4002870 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 >> v1:172.21.15.78:6789/0 conn(0x7ff3b4002870 legacy=0x7ff3b4004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.401+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 >> v1:172.21.15.78:6789/0 conn(0x7ff3b4002870 legacy=0x7ff3b4004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7ff3b4009e30 con 0x7ff3b40090f0 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 client.144279.objecter ms_handle_connect 0x7ff3b40090f0 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 client.144279.objecter resend_mon_ops 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 450634927 0 0) 0x7ff3b4009e30 con 0x7ff3b40090f0 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient(hunting): my global_id is 144285 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff3b40065d0 con 0x7ff3b40090f0 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1507680364 0 0) 0x7ff3b40065d0 con 0x7ff3b40090f0 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff3b4009e30 con 0x7ff3b40090f0 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff3c0005090 con 0x7ff3b40090f0 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient: dump: 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.427 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1211816572 0 0) 0x7ff3b4009e30 con 0x7ff3b40090f0 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 --> v1:172.21.15.78:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff3b400c400 con 0x7ff3b40090f0 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 <== mon.0 v1:172.21.15.78:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+4307 (unknown 0 0 3939407813) 0x7ff3c0007370 con 0x7ff3b40090f0 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 >> v1:172.21.15.78:6789/0 conn(0x7ff3b40090f0 legacy=0x7ff3b4009580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3cd7fa640 1 -- 172.21.15.78:0/2991392586 >> v1:172.21.15.78:6789/0 conn(0x7ff3b40090f0 legacy=0x7ff3b4009580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "leader", 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 80, 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 237304, 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:50:35.429 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.430 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.431 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:50:35.432 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.433 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.434 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.435 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.436 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:50:35.437 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: "osd": [ 2024-09-15T20:50:35.438 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 6 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: "mgr": [ 2024-09-15T20:50:35.439 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:50:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:50:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:50:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.440 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:50:35.440 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 10 librados: watch_flush enter 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 10 librados: watch_flush exit 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 20 client.144279.objecter shutdown clearing up homeless session... 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 10 client.144279.objecter successfully canceled tick 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 >> v1:172.21.15.146:6812/435598642 conn(0x7ff3b003e000 legacy=0x7ff3b00404c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 10 monclient: shutdown 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 >> v1:172.21.15.78:6789/0 conn(0x7ff3d0135b60 legacy=0x7ff3d01d1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d48c5640 1 -- 172.21.15.78:0/2991392586 reap_dead start 2024-09-15T20:50:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d48c5640 1 -- 172.21.15.78:0/2991392586 reap_dead start 2024-09-15T20:50:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 shutdown_connections 2024-09-15T20:50:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 >> 172.21.15.78:0/2991392586 conn(0x7ff3d00a63c0 msgr2=0x7ff3d013edc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 shutdown_connections 2024-09-15T20:50:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 1 -- 172.21.15.78:0/2991392586 wait complete. 2024-09-15T20:50:35.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.409+0000 7ff3d6328640 1 librados: shutdown 2024-09-15T20:50:35.443 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:50:35.443 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-09-15T20:50:35.443 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-09-15T20:50:35.558 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: get_monmap_and_config 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: monmap: 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:35.560968+0000 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:35.560968+0000 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:35.559 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 Processor -- start 2024-09-15T20:50:35.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- start start 2024-09-15T20:50:35.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: init 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient(hunting): picked mon.noname-c con 0x7fb144137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient(hunting): picked mon.noname-b con 0x7fb1440a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient(hunting): picked mon.noname-a con 0x7fb14413e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1440a6420 con 0x7fb14413e5f0 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1440a6700 con 0x7fb144137ce0 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1440a2110 con 0x7fb1440a9610 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient(hunting): authenticate will time out at 2296.445517s 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb143fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb1440a9610 0x7fb14413ced0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35278/0 (socket says 172.21.15.78:35278) 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1491bc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb14413e5f0 0x7fb1441409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46096/0 (socket says 172.21.15.78:46096) 2024-09-15T20:50:35.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb143fff640 1 -- 172.21.15.78:0/614433345 learned_addr learned my addr 172.21.15.78:0/614433345 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1603939539 0 0) 0x7fb1440a2110 con 0x7fb1440a9610 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient(hunting): my global_id is 154202 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb12c003530 con 0x7fb1440a9610 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1103557075 0 0) 0x7fb1440a6420 con 0x7fb14413e5f0 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient(hunting): my global_id is 144288 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1440a2110 con 0x7fb14413e5f0 2024-09-15T20:50:35.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2920774179 0 0) 0x7fb1440a2110 con 0x7fb14413e5f0 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 >> v1:172.21.15.78:6790/0 conn(0x7fb1440a9610 legacy=0x7fb14413ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 >> v1:172.21.15.146:6789/0 conn(0x7fb144137ce0 legacy=0x7fb1441380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 monclient: found mon.noname-a 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1440a4bc0 con 0x7fb14413e5f0 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb13c001a00 con 0x7fb14413e5f0 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: dump: 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 5 monclient: authenticate success, global_id 144288 2024-09-15T20:50:35.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb13c001ba0 con 0x7fb14413e5f0 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 1 -- 172.21.15.78:0/614433345 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb13c001fb0 con 0x7fb14413e5f0 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb1437fe640 10 monclient: dump: 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.572 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 4 set_mon_vals no callback set 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: shutdown 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- 172.21.15.78:0/614433345 >> v1:172.21.15.78:6789/0 conn(0x7fb14413e5f0 legacy=0x7fb1441409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- 172.21.15.78:0/614433345 shutdown_connections 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- 172.21.15.78:0/614433345 >> 172.21.15.78:0/614433345 conn(0x7fb1440aa6c0 msgr2=0x7fb1440aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- 172.21.15.78:0/614433345 shutdown_connections 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 -- 172.21.15.78:0/614433345 wait complete. 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 10 monclient: monmap: 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.573 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 librados: starting msgr at 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.557+0000 7fb14ac1f640 1 librados: starting objecter 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 Processor -- start 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 -- start start 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 librados: setting wanted keys 2024-09-15T20:50:35.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 librados: calling monclient init 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient: init 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient(hunting): picked mon.noname-a con 0x7fb1440a9610 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient(hunting): picked mon.noname-c con 0x7fb144137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient(hunting): picked mon.noname-b con 0x7fb14413e5f0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1440744a0 con 0x7fb1440a9610 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb14413c740 con 0x7fb14413e5f0 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1440a14b0 con 0x7fb144137ce0 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient(hunting): authenticate will time out at 2296.448643s 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb143fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb144137ce0 0x7fb1441d1960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35294/0 (socket says 172.21.15.78:35294) 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb143fff640 1 -- 172.21.15.78:0/2276426710 learned_addr learned my addr 172.21.15.78:0/2276426710 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1489bb640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb1440a9610 0x7fb14413dc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46100/0 (socket says 172.21.15.78:46100) 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.?.objecter ms_handle_connect 0x7fb144137ce0 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.?.objecter ms_handle_connect 0x7fb1440a9610 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1374966253 0 0) 0x7fb1440a14b0 con 0x7fb144137ce0 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): my global_id is 154205 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb130004ef0 con 0x7fb144137ce0 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 50585849 0 0) 0x7fb1440a6700 con 0x7fb1440a9610 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): my global_id is 144291 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1440a14b0 con 0x7fb1440a9610 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.?.objecter ms_handle_connect 0x7fb14413e5f0 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1937458730 0 0) 0x7fb130004ef0 con 0x7fb144137ce0 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1440a6700 con 0x7fb144137ce0 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb1380018a0 con 0x7fb144137ce0 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): dump: 2024-09-15T20:50:35.577 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1563680779 0 0) 0x7fb1440a14b0 con 0x7fb1440a9610 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb130004ef0 con 0x7fb1440a9610 2024-09-15T20:50:35.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb134001940 con 0x7fb1440a9610 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): dump: 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2451428327 0 0) 0x7fb14413c740 con 0x7fb14413e5f0 2024-09-15T20:50:35.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): my global_id is 164101 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1440a14b0 con 0x7fb14413e5f0 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2055762147 0 0) 0x7fb1440a6700 con 0x7fb144137ce0 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 >> v1:172.21.15.146:6789/0 conn(0x7fb14413e5f0 legacy=0x7fb1441d5090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 >> v1:172.21.15.78:6789/0 conn(0x7fb1440a9610 legacy=0x7fb14413dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 monclient: found mon.c 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb14413c460 con 0x7fb144137ce0 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 5 monclient: authenticate success, global_id 154205 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient: _renew_subs 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb14413b210 con 0x7fb144137ce0 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb1380047f0 con 0x7fb144137ce0 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb138004b40 con 0x7fb144137ce0 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 client.154205.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: dump: 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.581 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient: _renew_subs 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fb1441cc5e0 con 0x7fb144137ce0 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 librados: init done 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb142ffd640 4 set_mon_vals no callback set 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb14413c280 con 0x7fb14409ca20 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.154205.objecter ms_handle_connect 0x7fb14409ca20 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.154205.objecter resend_mon_ops 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb138010bb0 con 0x7fb144137ce0 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fb13804b4a0 con 0x7fb144137ce0 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.154205.objecter ms_dispatch 0x7fb144000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 3 client.154205.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 3 client.154205.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 20 client.154205.objecter dump_active .. 0 homeless 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1015448088 0 0) 0x7fb130004ef0 con 0x7fb14409ca20 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient(hunting): my global_id is 154208 2024-09-15T20:50:35.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1440a6700 con 0x7fb14409ca20 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3611008272 0 0) 0x7fb1440a6700 con 0x7fb14409ca20 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb130004ef0 con 0x7fb14409ca20 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb1340037c0 con 0x7fb14409ca20 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1491bc640 10 monclient: get_auth_request con 0x7fb13003e210 auth_method 0 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: dump: 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.584 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3912079504 0 0) 0x7fb130004ef0 con 0x7fb14409ca20 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1440a4160 con 0x7fb14409ca20 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 client.154205.objecter ms_handle_connect 0x7fb13003e210 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fb1340036b0 con 0x7fb14409ca20 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 >> v1:172.21.15.78:6790/0 conn(0x7fb14409ca20 legacy=0x7fb14409ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.561+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 >> v1:172.21.15.78:6790/0 conn(0x7fb14409ca20 legacy=0x7fb14409ce00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fb1440a0a00 con 0x7fb14409d510 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 client.154205.objecter ms_handle_connect 0x7fb14409d510 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 client.154205.objecter resend_mon_ops 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1326776948 0 0) 0x7fb1440a0a00 con 0x7fb14409d510 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient(hunting): my global_id is 154211 2024-09-15T20:50:35.587 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb130004ef0 con 0x7fb14409d510 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1328056974 0 0) 0x7fb130004ef0 con 0x7fb14409d510 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb1440a0a00 con 0x7fb14409d510 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb134003eb0 con 0x7fb14409d510 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient: dump: 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.588 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3182013622 0 0) 0x7fb1440a0a00 con 0x7fb14409d510 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 --> v1:172.21.15.78:6790/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb1441cc970 con 0x7fb14409d510 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 <== mon.2 v1:172.21.15.78:6790/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3971 (unknown 0 0 4214371833) 0x7fb1340040d0 con 0x7fb14409d510 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 >> v1:172.21.15.78:6790/0 conn(0x7fb14409d510 legacy=0x7fb1441da7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1417fa640 1 -- 172.21.15.78:0/2276426710 >> v1:172.21.15.78:6790/0 conn(0x7fb14409d510 legacy=0x7fb1441da7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:50:35.589 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 80, 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 123553, 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:50:35.590 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:50:35.591 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.592 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:50:35.593 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.594 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.595 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.596 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.597 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.598 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:50:35.599 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 2 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:50:35.600 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 10 librados: watch_flush enter 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 10 librados: watch_flush exit 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 20 client.154205.objecter shutdown clearing up homeless session... 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 10 client.154205.objecter successfully canceled tick 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 >> v1:172.21.15.146:6812/435598642 conn(0x7fb13003e210 legacy=0x7fb1300406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 10 monclient: shutdown 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 >> v1:172.21.15.78:6790/0 conn(0x7fb144137ce0 legacy=0x7fb1441d1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1491bc640 1 -- 172.21.15.78:0/2276426710 reap_dead start 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb1491bc640 1 -- 172.21.15.78:0/2276426710 reap_dead start 2024-09-15T20:50:35.602 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 shutdown_connections 2024-09-15T20:50:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 >> 172.21.15.78:0/2276426710 conn(0x7fb1440aa6c0 msgr2=0x7fb1441409e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 shutdown_connections 2024-09-15T20:50:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 1 -- 172.21.15.78:0/2276426710 wait complete. 2024-09-15T20:50:35.603 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.569+0000 7fb14ac1f640 1 librados: shutdown 2024-09-15T20:50:35.603 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:50:35.603 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-09-15T20:50:35.603 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-09-15T20:50:35.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient: get_monmap_and_config 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient: monmap: 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:35.717901+0000 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:35.717901+0000 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:35.716 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:35.717 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 1 Processor -- start 2024-09-15T20:50:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 1 -- start start 2024-09-15T20:50:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient: init 2024-09-15T20:50:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:50:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fc5f0137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fc5f00a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fc5f013e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00a6420 con 0x7fc5f013e5f0 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00a6700 con 0x7fc5f00a9610 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00a2110 con 0x7fc5f0137ce0 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f7d4b640 10 monclient(hunting): authenticate will time out at 2296.603155s 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f62e8640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fc5f013e5f0 0x7fc5f01409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46104/0 (socket says 172.21.15.78:46104) 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f62e8640 1 -- 172.21.15.78:0/1158283767 learned_addr learned my addr 172.21.15.78:0/1158283767 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f5ae7640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc5f0137ce0 0x7fc5f01380c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35316/0 (socket says 172.21.15.78:35316) 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1165119078 0 0) 0x7fc5f00a6420 con 0x7fc5f013e5f0 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient(hunting): my global_id is 144294 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5dc003530 con 0x7fc5f013e5f0 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2495323405 0 0) 0x7fc5f00a2110 con 0x7fc5f0137ce0 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient(hunting): my global_id is 154214 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f00a6420 con 0x7fc5f0137ce0 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2112672804 0 0) 0x7fc5dc003530 con 0x7fc5f013e5f0 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 >> v1:172.21.15.78:6790/0 conn(0x7fc5f0137ce0 legacy=0x7fc5f01380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 >> v1:172.21.15.146:6789/0 conn(0x7fc5f00a9610 legacy=0x7fc5f013ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 monclient: found mon.noname-a 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f00a4bc0 con 0x7fc5f013e5f0 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc5e80016a0 con 0x7fc5f013e5f0 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.713+0000 7fc5f4ae5640 10 monclient: dump: 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.723 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 5 monclient: authenticate success, global_id 144294 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5e80018c0 con 0x7fc5f013e5f0 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f4ae5640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f4ae5640 1 -- 172.21.15.78:0/1158283767 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc5e8001cd0 con 0x7fc5f013e5f0 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 4 set_mon_vals no callback set 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f4ae5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f4ae5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f4ae5640 10 monclient: dump: 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.724 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: shutdown 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/1158283767 >> v1:172.21.15.78:6789/0 conn(0x7fc5f013e5f0 legacy=0x7fc5f01409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/1158283767 shutdown_connections 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/1158283767 >> 172.21.15.78:0/1158283767 conn(0x7fc5f00aa6c0 msgr2=0x7fc5f00aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/1158283767 shutdown_connections 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/1158283767 wait complete. 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: monmap: 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 librados: starting msgr at 2024-09-15T20:50:35.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 librados: starting objecter 2024-09-15T20:50:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 Processor -- start 2024-09-15T20:50:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- start start 2024-09-15T20:50:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 librados: setting wanted keys 2024-09-15T20:50:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 librados: calling monclient init 2024-09-15T20:50:35.727 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: init 2024-09-15T20:50:35.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:50:35.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient(hunting): picked mon.noname-c con 0x7fc5f00a9610 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.732 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient(hunting): picked mon.noname-b con 0x7fc5f0137ce0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient(hunting): picked mon.noname-a con 0x7fc5f013e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f013c010 con 0x7fc5f013e5f0 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f013c2f0 con 0x7fc5f0137ce0 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5f00a14b0 con 0x7fc5f00a9610 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient(hunting): authenticate will time out at 2296.606108s 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f5ae7640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fc5f00a9610 0x7fc5f01cca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35328/0 (socket says 172.21.15.78:35328) 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f5ae7640 1 -- 172.21.15.78:0/957994936 learned_addr learned my addr 172.21.15.78:0/957994936 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 client.?.objecter ms_handle_connect 0x7fc5f00a9610 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 client.?.objecter ms_handle_connect 0x7fc5f013e5f0 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3808508333 0 0) 0x7fc5f00a14b0 con 0x7fc5f00a9610 2024-09-15T20:50:35.733 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): my global_id is 154217 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5d4004ec0 con 0x7fc5f00a9610 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836933505 0 0) 0x7fc5f013c010 con 0x7fc5f013e5f0 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): my global_id is 144297 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f00a14b0 con 0x7fc5f013e5f0 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 client.?.objecter ms_handle_connect 0x7fc5f0137ce0 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2171404036 0 0) 0x7fc5f00a14b0 con 0x7fc5f013e5f0 2024-09-15T20:50:35.734 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5f013c010 con 0x7fc5f013e5f0 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3101400780 0 0) 0x7fc5d4004ec0 con 0x7fc5f00a9610 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5f00a14b0 con 0x7fc5f00a9610 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc5e8001a70 con 0x7fc5f013e5f0 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): dump: 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.735 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc5ec001660 con 0x7fc5f00a9610 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): dump: 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.736 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1504599239 0 0) 0x7fc5f013c2f0 con 0x7fc5f0137ce0 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient(hunting): my global_id is 164107 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5d4004ec0 con 0x7fc5f0137ce0 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1537063987 0 0) 0x7fc5f013c010 con 0x7fc5f013e5f0 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 >> v1:172.21.15.78:6790/0 conn(0x7fc5f00a9610 legacy=0x7fc5f01cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 >> v1:172.21.15.146:6789/0 conn(0x7fc5f0137ce0 legacy=0x7fc5f01d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 monclient: found mon.a 2024-09-15T20:50:35.737 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5f0139f90 con 0x7fc5f013e5f0 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 5 monclient: authenticate success, global_id 144297 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: _renew_subs 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fc5f013ae30 con 0x7fc5f013e5f0 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 client.144297.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: _renew_subs 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fc5f01cd120 con 0x7fc5f013e5f0 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 librados: init done 2024-09-15T20:50:35.738 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fc5e8001820 con 0x7fc5f013e5f0 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc5e8004c00 con 0x7fc5f013e5f0 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e7fff640 4 set_mon_vals no callback set 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5e67fc640 10 monclient: dump: 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.739 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.717+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc5d80065d0 con 0x7fc5d8002870 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fc5e8010c70 con 0x7fc5f013e5f0 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 client.144297.objecter ms_handle_connect 0x7fc5d8002870 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 client.144297.objecter resend_mon_ops 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fc5e804b260 con 0x7fc5f013e5f0 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 client.144297.objecter ms_dispatch 0x7fc5f0000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 3 client.144297.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 3 client.144297.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 20 client.144297.objecter dump_active .. 0 homeless 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3967928543 0 0) 0x7fc5d80065d0 con 0x7fc5d8002870 2024-09-15T20:50:35.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient(hunting): my global_id is 164110 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5f013c010 con 0x7fc5d8002870 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5f52e6640 10 monclient: get_auth_request con 0x7fc5d403e270 auth_method 0 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 1497858013 0 0) 0x7fc5f013c010 con 0x7fc5d8002870 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5d80065d0 con 0x7fc5d8002870 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc5ec004f00 con 0x7fc5d8002870 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient: dump: 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.741 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 client.144297.objecter ms_handle_connect 0x7fc5d403e270 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 1454149087 0 0) 0x7fc5d80065d0 con 0x7fc5d8002870 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc5d800a170 con 0x7fc5d8002870 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (unknown 0 0 4045327437) 0x7fc5ec007260 con 0x7fc5d8002870 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient: handle_command_reply 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 >> v1:172.21.15.146:6789/0 conn(0x7fc5d8002870 legacy=0x7fc5d8004d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.721+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 >> v1:172.21.15.146:6789/0 conn(0x7fc5d8002870 legacy=0x7fc5d8004d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 10 start_mon_command cmd=[{"prefix": "mon_status"}] 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 1) -- 0x7fc5d8009e80 con 0x7fc5d8009140 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 client.144297.objecter ms_handle_connect 0x7fc5d8009140 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 client.144297.objecter resend_mon_ops 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1105647881 0 0) 0x7fc5d8009e80 con 0x7fc5d8009140 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient(hunting): my global_id is 164113 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc5d80065d0 con 0x7fc5d8009140 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 390+0+0 (unknown 3402511422 0 0) 0x7fc5d80065d0 con 0x7fc5d8009140 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fc5d8009e80 con 0x7fc5d8009140 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fc5ec0055f0 con 0x7fc5d8009140 2024-09-15T20:50:35.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient: dump: 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 206+0+0 (unknown 3732434793 0 0) 0x7fc5d8009e80 con 0x7fc5d8009140 2024-09-15T20:50:35.744 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 --> v1:172.21.15.146:6789/0 -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc5d800c420 con 0x7fc5d8009140 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 <== mon.1 v1:172.21.15.146:6789/0 5 ==== command_reply(tid 2: 0 ) ==== 8+0+3968 (unknown 0 0 2025222934) 0x7fc5ec006b20 con 0x7fc5d8009140 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient: handle_command_reply 2 [{"prefix": "mon_status"}] 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 >> v1:172.21.15.146:6789/0 conn(0x7fc5d8009140 legacy=0x7fc5d80095d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5e67fc640 1 -- 172.21.15.78:0/957994936 >> v1:172.21.15.146:6789/0 conn(0x7fc5d8009140 legacy=0x7fc5d80095d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "state": "peon", 2024-09-15T20:50:35.745 INFO:teuthology.orchestra.run.smithi078.stdout: "election_epoch": 80, 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "uptime": 775, 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum": [ 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: 0, 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: 1, 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: 2 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_age": 0, 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "required_con": "2450521708007866372", 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "required_mon": [ 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.746 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_con": "4540719139924082687", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "quorum_mon": [ 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.747 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "outside_quorum": [], 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "extra_probe_peers": [], 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "sync_provider": [], 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "monmap": { 2024-09-15T20:50:35.748 INFO:teuthology.orchestra.run.smithi078.stdout: "epoch": 1, 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "fsid": "a8a3eb37-3a92-4568-8aad-3536499064e4", 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "modified": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "created": "2024-09-15T20:31:01.119418Z", 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release": 19, 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "min_mon_release_name": "squid", 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "election_strategy": 1, 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "disallowed_leaders: ": "", 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false, 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "tiebreaker_mon": "", 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "removed_ranks: ": "", 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "features": { 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "persistent": [ 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "kraken", 2024-09-15T20:50:35.749 INFO:teuthology.orchestra.run.smithi078.stdout: "luminous", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "mimic", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "osdmap-prune", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "nautilus", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "octopus", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "pacific", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "elector-pinging", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "quincy", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "reef", 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "squid" 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "optional": [] 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "mons": [ 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.750 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 0, 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "a", 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789", 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6789/0", 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6789/0", 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.751 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 1, 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "b", 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789", 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.752 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.146:6789/0", 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.146:6789/0", 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "rank": 2, 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "name": "c", 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addrs": { 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "addrvec": [ 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "type": "v1", 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790", 2024-09-15T20:50:35.753 INFO:teuthology.orchestra.run.smithi078.stdout: "nonce": 0 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "addr": "172.21.15.78:6790/0", 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "public_addr": "172.21.15.78:6790/0", 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "priority": 0, 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "weight": 0, 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "crush_location": "{}" 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "feature_map": { 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "mon": [ 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.754 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: ], 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "client": [ 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: { 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "features": "0x3f03dffffffdffff", 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "release": "squid", 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "num": 1 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: } 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: ] 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: }, 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout: "stretch_mode": false 2024-09-15T20:50:35.755 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 10 librados: watch_flush enter 2024-09-15T20:50:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 10 librados: watch_flush exit 2024-09-15T20:50:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 20 client.144297.objecter shutdown clearing up homeless session... 2024-09-15T20:50:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 10 client.144297.objecter successfully canceled tick 2024-09-15T20:50:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 >> v1:172.21.15.146:6812/435598642 conn(0x7fc5d403e270 legacy=0x7fc5d4040730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 10 monclient: shutdown 2024-09-15T20:50:35.757 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 >> v1:172.21.15.78:6789/0 conn(0x7fc5f013e5f0 legacy=0x7fc5f01d4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f62e8640 1 -- 172.21.15.78:0/957994936 reap_dead start 2024-09-15T20:50:35.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f62e8640 1 -- 172.21.15.78:0/957994936 reap_dead start 2024-09-15T20:50:35.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 shutdown_connections 2024-09-15T20:50:35.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 >> 172.21.15.78:0/957994936 conn(0x7fc5f00aa6c0 msgr2=0x7fc5f01409e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 shutdown_connections 2024-09-15T20:50:35.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 1 -- 172.21.15.78:0/957994936 wait complete. 2024-09-15T20:50:35.758 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.729+0000 7fc5f7d4b640 1 librados: shutdown 2024-09-15T20:50:35.758 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-09-15T20:50:35.758 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-09-15T20:50:35.759 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-09-15T20:50:35.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.857+0000 7f5d5400b640 10 monclient: get_monmap_and_config 2024-09-15T20:50:35.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.857+0000 7f5d5400b640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.861 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.857+0000 7f5d5400b640 10 monclient: monmap: 2024-09-15T20:50:35.861 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:35.863712+0000 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:35.863712+0000 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:35.862 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.857+0000 7f5d5400b640 1 Processor -- start 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.857+0000 7f5d5400b640 1 -- start start 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.857+0000 7f5d5400b640 10 monclient: init 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4c135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4c13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c0745e0 con 0x7f5d4c135b60 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c06ecf0 con 0x7f5d4c0a24e0 2024-09-15T20:50:35.870 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c06efd0 con 0x7f5d4c13c220 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): authenticate will time out at 2296.748288s 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d51da7640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5d4c135b60 0x7f5d4c135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46124/0 (socket says 172.21.15.78:46124) 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d51da7640 1 -- 172.21.15.78:0/1292253220 learned_addr learned my addr 172.21.15.78:0/1292253220 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 470070423 0 0) 0x7f5d4c0745e0 con 0x7f5d4c135b60 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient(hunting): my global_id is 144300 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d34003530 con 0x7f5d4c135b60 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 231989427 0 0) 0x7f5d4c06efd0 con 0x7f5d4c13c220 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.871 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient(hunting): my global_id is 154220 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d4c0745e0 con 0x7f5d4c13c220 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1732448884 0 0) 0x7f5d34003530 con 0x7f5d4c135b60 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 >> v1:172.21.15.78:6790/0 conn(0x7f5d4c13c220 legacy=0x7f5d4c13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 >> v1:172.21.15.146:6789/0 conn(0x7f5d4c0a24e0 legacy=0x7f5d4c13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 monclient: found mon.noname-a 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c136650 con 0x7f5d4c135b60 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.872 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5d440016a0 con 0x7f5d4c135b60 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: dump: 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.873 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 5 monclient: authenticate success, global_id 144300 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d44001840 con 0x7f5d4c135b60 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 1 -- 172.21.15.78:0/1292253220 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5d44001c50 con 0x7f5d4c135b60 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 4 set_mon_vals no callback set 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d50da5640 10 monclient: dump: 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.874 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: shutdown 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- 172.21.15.78:0/1292253220 >> v1:172.21.15.78:6789/0 conn(0x7f5d4c135b60 legacy=0x7f5d4c135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- 172.21.15.78:0/1292253220 shutdown_connections 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- 172.21.15.78:0/1292253220 >> 172.21.15.78:0/1292253220 conn(0x7f5d4c0a63c0 msgr2=0x7f5d4c0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- 172.21.15.78:0/1292253220 shutdown_connections 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- 172.21.15.78:0/1292253220 wait complete. 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: build_initial_monmap 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: monmap: 2024-09-15T20:50:35.875 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 librados: starting msgr at 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 librados: starting objecter 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 Processor -- start 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- start start 2024-09-15T20:50:35.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 librados: setting wanted keys 2024-09-15T20:50:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 librados: calling monclient init 2024-09-15T20:50:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: init 2024-09-15T20:50:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): picked mon.noname-b con 0x7f5d4c0a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): picked mon.noname-c con 0x7f5d4c135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): picked mon.noname-a con 0x7f5d4c13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:35.877 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c073620 con 0x7f5d4c13c220 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c0a3780 con 0x7f5d4c0a24e0 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5d4c09d180 con 0x7f5d4c135b60 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient(hunting): authenticate will time out at 2296.750993s 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d515a6640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f5d4c135b60 0x7f5d4c072520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35348/0 (socket says 172.21.15.78:35348) 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d515a6640 1 -- 172.21.15.78:0/786906520 learned_addr learned my addr 172.21.15.78:0/786906520 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 client.?.objecter ms_handle_connect 0x7f5d4c135b60 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.878 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 client.?.objecter ms_handle_connect 0x7f5d4c13c220 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 661655745 0 0) 0x7f5d4c09d180 con 0x7f5d4c135b60 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): my global_id is 154223 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d38004ec0 con 0x7f5d4c135b60 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2662596646 0 0) 0x7f5d4c073620 con 0x7f5d4c13c220 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.879 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): my global_id is 144303 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d4c09d180 con 0x7f5d4c13c220 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 client.?.objecter ms_handle_connect 0x7f5d4c0a24e0 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3580812635 0 0) 0x7f5d38004ec0 con 0x7f5d4c135b60 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d4c073620 con 0x7f5d4c135b60 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5d40001dc0 con 0x7f5d4c135b60 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): dump: 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.880 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2923455733 0 0) 0x7f5d4c09d180 con 0x7f5d4c13c220 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5d38004ec0 con 0x7f5d4c13c220 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5d3c001830 con 0x7f5d4c13c220 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): dump: 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.881 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2575417430 0 0) 0x7f5d4c0a3780 con 0x7f5d4c0a24e0 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient(hunting): my global_id is 164119 2024-09-15T20:50:35.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5d4c09d180 con 0x7f5d4c0a24e0 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2044913881 0 0) 0x7f5d4c073620 con 0x7f5d4c135b60 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 >> v1:172.21.15.146:6789/0 conn(0x7f5d4c0a24e0 legacy=0x7f5d4c071e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 >> v1:172.21.15.78:6789/0 conn(0x7f5d4c13c220 legacy=0x7f5d4c1d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 monclient: found mon.c 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4c138670 con 0x7f5d4c135b60 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient: _finish_auth 0 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 5 monclient: authenticate success, global_id 154223 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: _renew_subs 2024-09-15T20:50:35.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5d4c136e70 con 0x7f5d4c135b60 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 client.154223.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: _renew_subs 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f5d4c072c30 con 0x7f5d4c135b60 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.861+0000 7f5d5400b640 1 librados: init done 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5d400020c0 con 0x7f5d4c135b60 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5d40003df0 con 0x7f5d4c135b60 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4bfff640 4 set_mon_vals no callback set 2024-09-15T20:50:35.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 10 monclient: dump: 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d5400b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d5400b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d5400b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:35.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2c005090 con 0x7f5d4c135b60 2024-09-15T20:50:35.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f5d40003f90 con 0x7f5d4c135b60 2024-09-15T20:50:35.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f5d4004b940 con 0x7f5d4c135b60 2024-09-15T20:50:35.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 10 client.154223.objecter ms_dispatch 0x7f5d4c000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:35.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 3 client.154223.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:35.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 3 client.154223.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:35.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 20 client.154223.objecter dump_active .. 0 homeless 2024-09-15T20:50:35.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d51da7640 10 monclient: get_auth_request con 0x7f5d3803e240 auth_method 0 2024-09-15T20:50:35.886 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.865+0000 7f5d4a7fc640 10 client.154223.objecter ms_handle_connect 0x7f5d3803e240 2024-09-15T20:50:35.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.869+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5d40016ea0 con 0x7f5d4c135b60 2024-09-15T20:50:35.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.869+0000 7f5d4a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:35.887 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:35.869+0000 7f5d4a7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:36.101 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.097+0000 7f5d5400b640 10 start_mon_command cmd=[{"prefix": "mon scrub"}] 2024-09-15T20:50:36.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.097+0000 7f5d5400b640 10 monclient: _send_command 2 [{"prefix": "mon scrub"}] 2024-09-15T20:50:36.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.097+0000 7f5d5400b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:36.102 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.097+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f5d2c005c50 con 0x7f5d4c135b60 2024-09-15T20:50:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d4a7fc640 1 -- 172.21.15.78:0/786906520 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (unknown 246732539 0 0) 0x7f5d400167a0 con 0x7f5d4c135b60 2024-09-15T20:50:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d4a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "mon scrub"}] 2024-09-15T20:50:36.103 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d4a7fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:50:36.104 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.104 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=499786549,config=2586702516,health=797083738,kv=199072906,logm=436288622}) 2024-09-15T20:50:36.104 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {auth=25,config=2,health=12,kv=11,logm=50} crc {auth=1064116750,config=2586702516,health=314794474,kv=3449721867,logm=162461184}) 2024-09-15T20:50:36.104 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.104 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {auth=24,config=2,health=11,kv=12,logm=51} crc {auth=499786549,config=2586702516,health=797083738,kv=199072906,logm=436288622}) 2024-09-15T20:50:36.104 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {auth=25,config=2,health=10,kv=12,logm=51} crc {auth=4050929310,config=2586702516,health=2531951926,kv=1721617520,logm=3829919344}) 2024-09-15T20:50:36.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 10 librados: watch_flush enter 2024-09-15T20:50:36.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 10 librados: watch_flush exit 2024-09-15T20:50:36.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 20 client.154223.objecter shutdown clearing up homeless session... 2024-09-15T20:50:36.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 10 client.154223.objecter successfully canceled tick 2024-09-15T20:50:36.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 >> v1:172.21.15.146:6812/435598642 conn(0x7f5d3803e240 legacy=0x7f5d38040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:36.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 10 monclient: shutdown 2024-09-15T20:50:36.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:36.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 >> v1:172.21.15.78:6790/0 conn(0x7f5d4c135b60 legacy=0x7f5d4c072520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:36.107 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.107 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3691130586}) 2024-09-15T20:50:36.107 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=103356420}) 2024-09-15T20:50:36.107 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.107 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3691130586}) 2024-09-15T20:50:36.107 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.101+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2216694818}) 2024-09-15T20:50:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 shutdown_connections 2024-09-15T20:50:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.101+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 >> 172.21.15.78:0/786906520 conn(0x7f5d4c0a63c0 msgr2=0x7f5d4c12ea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.105+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 shutdown_connections 2024-09-15T20:50:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.105+0000 7f5d5400b640 1 -- 172.21.15.78:0/786906520 wait complete. 2024-09-15T20:50:36.108 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:36.105+0000 7f5d5400b640 1 librados: shutdown 2024-09-15T20:50:36.109 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.109 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236928762}) 2024-09-15T20:50:36.109 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1782171137}) 2024-09-15T20:50:36.109 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.109 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2236928762}) 2024-09-15T20:50:36.109 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=238811664}) 2024-09-15T20:50:36.110 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.110 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299632487}) 2024-09-15T20:50:36.110 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3127922100}) 2024-09-15T20:50:36.110 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.110 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2299632487}) 2024-09-15T20:50:36.110 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.105+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4028536198}) 2024-09-15T20:50:36.112 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.112 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993796072}) 2024-09-15T20:50:36.112 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4012581192}) 2024-09-15T20:50:36.112 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.112 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2993796072}) 2024-09-15T20:50:36.112 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1762128763}) 2024-09-15T20:50:36.113 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.113 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477360601}) 2024-09-15T20:50:36.114 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2852734538}) 2024-09-15T20:50:36.114 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.114 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=477360601}) 2024-09-15T20:50:36.114 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.109+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=857129210}) 2024-09-15T20:50:36.115 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.115 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248511414}) 2024-09-15T20:50:36.115 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=661537704}) 2024-09-15T20:50:36.115 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.115 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2248511414}) 2024-09-15T20:50:36.115 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4109924745}) 2024-09-15T20:50:36.117 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.117 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972022482}) 2024-09-15T20:50:36.117 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3266432484}) 2024-09-15T20:50:36.117 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.117 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=972022482}) 2024-09-15T20:50:36.117 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.113+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1109224681}) 2024-09-15T20:50:36.121 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.121 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246477186}) 2024-09-15T20:50:36.121 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1646273423}) 2024-09-15T20:50:36.121 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.121 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3246477186}) 2024-09-15T20:50:36.121 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1358127148}) 2024-09-15T20:50:36.121 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.121 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145195909}) 2024-09-15T20:50:36.122 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1703906326}) 2024-09-15T20:50:36.122 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.122 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1145195909}) 2024-09-15T20:50:36.122 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=826950930}) 2024-09-15T20:50:36.122 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.122 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276005064}) 2024-09-15T20:50:36.122 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2642193032}) 2024-09-15T20:50:36.122 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.123 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3276005064}) 2024-09-15T20:50:36.123 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.117+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1251965942}) 2024-09-15T20:50:36.124 INFO:tasks.mon_thrash.mon_thrasher:waiting for 1.0 secs before continuing thrashing 2024-09-15T20:50:36.124 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.124 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3323147027}) 2024-09-15T20:50:36.124 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2522184026}) 2024-09-15T20:50:36.125 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.125 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3323147027}) 2024-09-15T20:50:36.125 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=629325246}) 2024-09-15T20:50:36.125 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.125 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4023743981}) 2024-09-15T20:50:36.125 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3686414866}) 2024-09-15T20:50:36.125 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.126 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4023743981}) 2024-09-15T20:50:36.126 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.121+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2944534101}) 2024-09-15T20:50:36.127 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.127 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192718505}) 2024-09-15T20:50:36.127 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3134808705}) 2024-09-15T20:50:36.127 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.127 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4192718505}) 2024-09-15T20:50:36.127 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1973731896}) 2024-09-15T20:50:36.129 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.129 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326727689}) 2024-09-15T20:50:36.129 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=726875530}) 2024-09-15T20:50:36.129 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.129 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1326727689}) 2024-09-15T20:50:36.129 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2323344688}) 2024-09-15T20:50:36.131 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.131 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.125+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161343827}) 2024-09-15T20:50:36.131 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2825881883}) 2024-09-15T20:50:36.131 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.131 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4161343827}) 2024-09-15T20:50:36.131 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2271935640}) 2024-09-15T20:50:36.132 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.132 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=149094568}) 2024-09-15T20:50:36.132 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=783306238}) 2024-09-15T20:50:36.132 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.133 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=149094568}) 2024-09-15T20:50:36.133 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3905659712}) 2024-09-15T20:50:36.134 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.134 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982033753}) 2024-09-15T20:50:36.134 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3122101653}) 2024-09-15T20:50:36.134 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.134 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2982033753}) 2024-09-15T20:50:36.134 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.129+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=645832173}) 2024-09-15T20:50:36.136 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.136 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404696731}) 2024-09-15T20:50:36.136 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1475539508}) 2024-09-15T20:50:36.136 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.136 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2404696731}) 2024-09-15T20:50:36.136 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2017840225}) 2024-09-15T20:50:36.138 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.138 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139383604}) 2024-09-15T20:50:36.138 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3741824878}) 2024-09-15T20:50:36.138 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.138 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2139383604}) 2024-09-15T20:50:36.138 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.133+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2768252031}) 2024-09-15T20:50:36.140 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.140 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=442124770}) 2024-09-15T20:50:36.140 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1125073334}) 2024-09-15T20:50:36.140 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.140 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=442124770}) 2024-09-15T20:50:36.140 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1237347303}) 2024-09-15T20:50:36.142 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.142 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669227106}) 2024-09-15T20:50:36.142 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2753840487}) 2024-09-15T20:50:36.142 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.142 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=669227106}) 2024-09-15T20:50:36.142 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.137+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3225905179}) 2024-09-15T20:50:36.143 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.144 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538615131}) 2024-09-15T20:50:36.144 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=671417987}) 2024-09-15T20:50:36.144 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.144 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2538615131}) 2024-09-15T20:50:36.144 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=612133404}) 2024-09-15T20:50:36.145 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.145 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042717735}) 2024-09-15T20:50:36.145 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1639023418}) 2024-09-15T20:50:36.145 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.145 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2042717735}) 2024-09-15T20:50:36.146 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.141+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1760327709}) 2024-09-15T20:50:36.147 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.147 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=461906533}) 2024-09-15T20:50:36.147 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=602759614}) 2024-09-15T20:50:36.147 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.147 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=461906533}) 2024-09-15T20:50:36.147 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4168960613}) 2024-09-15T20:50:36.149 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.149 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186916562}) 2024-09-15T20:50:36.149 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2815155592}) 2024-09-15T20:50:36.149 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.149 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4186916562}) 2024-09-15T20:50:36.149 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.145+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3693346637}) 2024-09-15T20:50:36.151 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.151 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085368816}) 2024-09-15T20:50:36.151 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1007841111}) 2024-09-15T20:50:36.151 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.151 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4085368816}) 2024-09-15T20:50:36.151 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1219890595}) 2024-09-15T20:50:36.153 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.153 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176132748}) 2024-09-15T20:50:36.153 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1534640594}) 2024-09-15T20:50:36.153 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.153 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2176132748}) 2024-09-15T20:50:36.153 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.149+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1370554908}) 2024-09-15T20:50:36.154 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.154 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343191583}) 2024-09-15T20:50:36.155 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2347498647}) 2024-09-15T20:50:36.155 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.155 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=343191583}) 2024-09-15T20:50:36.155 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3549074799}) 2024-09-15T20:50:36.156 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.156 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230056457}) 2024-09-15T20:50:36.156 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2548456215}) 2024-09-15T20:50:36.157 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.157 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=230056457}) 2024-09-15T20:50:36.157 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3941838845}) 2024-09-15T20:50:36.158 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.158 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252333389}) 2024-09-15T20:50:36.158 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3255542020}) 2024-09-15T20:50:36.158 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.158 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3252333389}) 2024-09-15T20:50:36.158 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.153+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1854429939}) 2024-09-15T20:50:36.160 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.160 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705446690}) 2024-09-15T20:50:36.160 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1401370219}) 2024-09-15T20:50:36.160 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.160 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1705446690}) 2024-09-15T20:50:36.160 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1296517555}) 2024-09-15T20:50:36.161 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.161 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479214676}) 2024-09-15T20:50:36.161 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=268834650}) 2024-09-15T20:50:36.161 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.162 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1479214676}) 2024-09-15T20:50:36.162 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.157+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2229640530}) 2024-09-15T20:50:36.163 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.163 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3134932841}) 2024-09-15T20:50:36.163 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1922010465}) 2024-09-15T20:50:36.163 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.163 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3134932841}) 2024-09-15T20:50:36.163 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=624867193}) 2024-09-15T20:50:36.165 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.165 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302456303}) 2024-09-15T20:50:36.165 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1786152436}) 2024-09-15T20:50:36.165 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.165 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=302456303}) 2024-09-15T20:50:36.165 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.161+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2734799576}) 2024-09-15T20:50:36.167 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.167 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650506777}) 2024-09-15T20:50:36.167 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1115289735}) 2024-09-15T20:50:36.167 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.167 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=650506777}) 2024-09-15T20:50:36.167 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3192982221}) 2024-09-15T20:50:36.168 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.168 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176379254}) 2024-09-15T20:50:36.168 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=937059010}) 2024-09-15T20:50:36.168 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.169 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1176379254}) 2024-09-15T20:50:36.169 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3856030283}) 2024-09-15T20:50:36.170 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.170 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680432228}) 2024-09-15T20:50:36.170 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1567503286}) 2024-09-15T20:50:36.170 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.170 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3680432228}) 2024-09-15T20:50:36.170 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.165+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3790382215}) 2024-09-15T20:50:36.171 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.172 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249880517}) 2024-09-15T20:50:36.172 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3638870590}) 2024-09-15T20:50:36.172 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.172 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1249880517}) 2024-09-15T20:50:36.172 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2594140396}) 2024-09-15T20:50:36.173 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.173 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058331087}) 2024-09-15T20:50:36.173 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3579932128}) 2024-09-15T20:50:36.173 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.173 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4058331087}) 2024-09-15T20:50:36.173 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.169+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3052135333}) 2024-09-15T20:50:36.175 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.175 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759407333}) 2024-09-15T20:50:36.175 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1484900878}) 2024-09-15T20:50:36.175 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.175 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=759407333}) 2024-09-15T20:50:36.175 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3389361447}) 2024-09-15T20:50:36.176 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.176 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195478340}) 2024-09-15T20:50:36.177 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1039751689}) 2024-09-15T20:50:36.177 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.177 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4195478340}) 2024-09-15T20:50:36.177 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4097997645}) 2024-09-15T20:50:36.178 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.178 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233761639}) 2024-09-15T20:50:36.178 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=235402865}) 2024-09-15T20:50:36.178 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.179 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1233761639}) 2024-09-15T20:50:36.179 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.173+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2248572534}) 2024-09-15T20:50:36.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2148305966}) 2024-09-15T20:50:36.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3375720375}) 2024-09-15T20:50:36.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2148305966}) 2024-09-15T20:50:36.180 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=506874958}) 2024-09-15T20:50:36.181 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.181 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616304449}) 2024-09-15T20:50:36.182 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2451634280}) 2024-09-15T20:50:36.182 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.182 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1616304449}) 2024-09-15T20:50:36.182 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.177+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3117655572}) 2024-09-15T20:50:36.183 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.183 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412939199}) 2024-09-15T20:50:36.183 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3346161444}) 2024-09-15T20:50:36.184 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.184 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2412939199}) 2024-09-15T20:50:36.184 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1788368022}) 2024-09-15T20:50:36.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171959876}) 2024-09-15T20:50:36.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2828820814}) 2024-09-15T20:50:36.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1171959876}) 2024-09-15T20:50:36.185 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.181+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3721218811}) 2024-09-15T20:50:36.187 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.187 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240586314}) 2024-09-15T20:50:36.187 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3130472419}) 2024-09-15T20:50:36.187 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.187 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4240586314}) 2024-09-15T20:50:36.187 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1699028639}) 2024-09-15T20:50:36.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773441223}) 2024-09-15T20:50:36.188 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1615851217}) 2024-09-15T20:50:36.189 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.189 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3773441223}) 2024-09-15T20:50:36.189 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2013970335}) 2024-09-15T20:50:36.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=539578694}) 2024-09-15T20:50:36.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3587637083}) 2024-09-15T20:50:36.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.190 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=539578694}) 2024-09-15T20:50:36.191 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.185+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1018515230}) 2024-09-15T20:50:36.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350180862}) 2024-09-15T20:50:36.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2450966585}) 2024-09-15T20:50:36.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3350180862}) 2024-09-15T20:50:36.192 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2945446967}) 2024-09-15T20:50:36.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027646895}) 2024-09-15T20:50:36.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=270442101}) 2024-09-15T20:50:36.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1027646895}) 2024-09-15T20:50:36.194 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.189+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3563241908}) 2024-09-15T20:50:36.195 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728640383}) 2024-09-15T20:50:36.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=979490891}) 2024-09-15T20:50:36.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1728640383}) 2024-09-15T20:50:36.196 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2929181579}) 2024-09-15T20:50:36.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922970830}) 2024-09-15T20:50:36.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2237175629}) 2024-09-15T20:50:36.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3922970830}) 2024-09-15T20:50:36.198 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.193+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=477435118}) 2024-09-15T20:50:36.199 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.199 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639059746}) 2024-09-15T20:50:36.199 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=4288129275}) 2024-09-15T20:50:36.199 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.199 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3639059746}) 2024-09-15T20:50:36.199 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2967717738}) 2024-09-15T20:50:36.200 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1874073653}) 2024-09-15T20:50:36.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=744197165}) 2024-09-15T20:50:36.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1874073653}) 2024-09-15T20:50:36.201 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2443877985}) 2024-09-15T20:50:36.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643766110}) 2024-09-15T20:50:36.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=370629254}) 2024-09-15T20:50:36.202 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.203 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=643766110}) 2024-09-15T20:50:36.203 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.197+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3542508327}) 2024-09-15T20:50:36.204 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.204 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021182522}) 2024-09-15T20:50:36.204 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3443265448}) 2024-09-15T20:50:36.204 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.204 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=4021182522}) 2024-09-15T20:50:36.205 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3059149551}) 2024-09-15T20:50:36.206 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.206 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265305048}) 2024-09-15T20:50:36.206 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1148882283}) 2024-09-15T20:50:36.206 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.206 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2265305048}) 2024-09-15T20:50:36.206 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.201+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2598836253}) 2024-09-15T20:50:36.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.207 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810561932}) 2024-09-15T20:50:36.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3820223643}) 2024-09-15T20:50:36.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2810561932}) 2024-09-15T20:50:36.208 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=335992707}) 2024-09-15T20:50:36.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242383032}) 2024-09-15T20:50:36.209 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3386533102}) 2024-09-15T20:50:36.210 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.210 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2242383032}) 2024-09-15T20:50:36.210 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.205+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4265089670}) 2024-09-15T20:50:36.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075988721}) 2024-09-15T20:50:36.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3162401314}) 2024-09-15T20:50:36.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2075988721}) 2024-09-15T20:50:36.211 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2532625993}) 2024-09-15T20:50:36.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1619433280}) 2024-09-15T20:50:36.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3860669309}) 2024-09-15T20:50:36.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=1619433280}) 2024-09-15T20:50:36.213 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.209+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2208016436}) 2024-09-15T20:50:36.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654093178}) 2024-09-15T20:50:36.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1565035035}) 2024-09-15T20:50:36.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2654093178}) 2024-09-15T20:50:36.215 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2964069505}) 2024-09-15T20:50:36.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584448298}) 2024-09-15T20:50:36.216 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=351610842}) 2024-09-15T20:50:36.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=584448298}) 2024-09-15T20:50:36.217 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1467575455}) 2024-09-15T20:50:36.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158448169}) 2024-09-15T20:50:36.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=2128731477}) 2024-09-15T20:50:36.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2158448169}) 2024-09-15T20:50:36.218 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.213+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=3710734827}) 2024-09-15T20:50:36.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3932858184}) 2024-09-15T20:50:36.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3594329534}) 2024-09-15T20:50:36.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3932858184}) 2024-09-15T20:50:36.220 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=4024625058}) 2024-09-15T20:50:36.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687152375}) 2024-09-15T20:50:36.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1819793503}) 2024-09-15T20:50:36.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=687152375}) 2024-09-15T20:50:36.221 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.217+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2018722176}) 2024-09-15T20:50:36.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947422000}) 2024-09-15T20:50:36.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3599740802}) 2024-09-15T20:50:36.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=947422000}) 2024-09-15T20:50:36.223 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1988853627}) 2024-09-15T20:50:36.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380061299}) 2024-09-15T20:50:36.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=8956748}) 2024-09-15T20:50:36.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2380061299}) 2024-09-15T20:50:36.225 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2991922708}) 2024-09-15T20:50:36.226 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.226 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165704561}) 2024-09-15T20:50:36.226 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3322818110}) 2024-09-15T20:50:36.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=2165704561}) 2024-09-15T20:50:36.227 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.221+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=446702126}) 2024-09-15T20:50:36.228 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.228 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846443980}) 2024-09-15T20:50:36.228 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1842680185}) 2024-09-15T20:50:36.228 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.228 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3846443980}) 2024-09-15T20:50:36.229 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=1948611086}) 2024-09-15T20:50:36.230 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.230 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740031276}) 2024-09-15T20:50:36.230 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=569063857}) 2024-09-15T20:50:36.230 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.230 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=3740031276}) 2024-09-15T20:50:36.230 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.225+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2510148559}) 2024-09-15T20:50:36.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166693202}) 2024-09-15T20:50:36.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=1925769851}) 2024-09-15T20:50:36.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=166693202}) 2024-09-15T20:50:36.232 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2715167016}) 2024-09-15T20:50:36.233 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.233 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231551784}) 2024-09-15T20:50:36.233 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=100} crc {logm=3328525653}) 2024-09-15T20:50:36.233 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.234 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=100} crc {logm=231551784}) 2024-09-15T20:50:36.234 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.229+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=100} crc {logm=2880234052}) 2024-09-15T20:50:36.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=32} crc {logm=329070209,mdsmap=215940802,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=3146500049,mon_config_key=3082965474,monmap=3785329752,nvmeofgw=598337686,osd_metadata=2781288008,osd_pg_creating=2706592551,osd_snap=1835049866}) 2024-09-15T20:50:36.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {logm=17,mdsmap=3,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=12,mon_config_key=17,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=28} crc {logm=2269487704,mdsmap=503983409,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=1025495048,mon_config_key=3551184854,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=2706592551,osd_snap=3958357480}) 2024-09-15T20:50:36.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {logm=19,mdsmap=2,mgr=8,mgr_command_descs=1,mgr_metadata=1,mgrstat=9,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=5,osd_pg_creating=1,osd_snap=32} crc {logm=329070209,mdsmap=215940802,mgr=2617012046,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=3146500049,mon_config_key=3082965474,monmap=3785329752,nvmeofgw=598337686,osd_metadata=2781288008,osd_pg_creating=2706592551,osd_snap=1835049866}) 2024-09-15T20:50:36.236 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {logm=19,mdsmap=3,mgr=7,mgr_command_descs=1,mgr_metadata=1,mgrstat=11,mon_config_key=16,monmap=3,nvmeofgw=3,osd_metadata=6,osd_pg_creating=1,osd_snap=29} crc {logm=329070209,mdsmap=503983409,mgr=2467901132,mgr_command_descs=2551042219,mgr_metadata=1997488951,mgrstat=650625805,mon_config_key=1408905361,monmap=3785329752,nvmeofgw=598337686,osd_metadata=1681388020,osd_pg_creating=2706592551,osd_snap=3388565827}) 2024-09-15T20:50:36.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=1233986335,osdmap=2692983684}) 2024-09-15T20:50:36.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=1233986335,osdmap=3087234217}) 2024-09-15T20:50:36.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=1233986335,osdmap=2692983684}) 2024-09-15T20:50:36.238 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.233+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osd_snap=7,osdmap=93} crc {osd_snap=1233986335,osdmap=2991323681}) 2024-09-15T20:50:36.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1387579815}) 2024-09-15T20:50:36.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1981453933}) 2024-09-15T20:50:36.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=1387579815}) 2024-09-15T20:50:36.240 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=209111090}) 2024-09-15T20:50:36.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=443693607}) 2024-09-15T20:50:36.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=2903594055}) 2024-09-15T20:50:36.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=443693607}) 2024-09-15T20:50:36.242 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.237+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=1459048396}) 2024-09-15T20:50:36.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.241+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.241+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4225451832}) 2024-09-15T20:50:36.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.241+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=100} crc {osdmap=1981429029}) 2024-09-15T20:50:36.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.241+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.241+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=100} crc {osdmap=4225451832}) 2024-09-15T20:50:36.246 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.241+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=100} crc {osdmap=3229686282}) 2024-09-15T20:50:36.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.245+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.248 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.245+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=3548645333}) 2024-09-15T20:50:36.249 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.245+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.1 ScrubResult(keys {osdmap=83} crc {osdmap=328939398}) 2024-09-15T20:50:36.249 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.245+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : scrub mismatch 2024-09-15T20:50:36.249 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.245+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.0 ScrubResult(keys {osdmap=81} crc {osdmap=3548645333}) 2024-09-15T20:50:36.249 INFO:tasks.ceph.mon.a.smithi078.stderr:2024-09-15T20:50:36.245+0000 7fa8de033640 -1 log_channel(cluster) log [ERR] : mon.2 ScrubResult(keys {osdmap=83} crc {osdmap=3734458403}) 2024-09-15T20:50:37.124 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-09-15T20:50:37.125 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient: get_monmap_and_config 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient: build_initial_monmap 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient: monmap: 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:37.258105+0000 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:37.258105+0000 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.256 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:37.257 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:37.257 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:37.257 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.263 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 1 Processor -- start 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 1 -- start start 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient: init 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient(hunting): picked mon.noname-b con 0x7f6450135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient(hunting): picked mon.noname-c con 0x7f64500a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient(hunting): picked mon.noname-a con 0x7f645013c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64500745e0 con 0x7f645013c220 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f645006ecf0 con 0x7f64500a24e0 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f645006efd0 con 0x7f6450135b60 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:37.264 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 10 monclient(hunting): authenticate will time out at 2298.142988s 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f64562f3640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f645013c220 0x7f645013e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46136/0 (socket says 172.21.15.78:46136) 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6455af2640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f6450135b60 0x7f6450135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35362/0 (socket says 172.21.15.78:35362) 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f64562f3640 1 -- 172.21.15.78:0/1051781023 learned_addr learned my addr 172.21.15.78:0/1051781023 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2532719701 0 0) 0x7f645006efd0 con 0x7f6450135b60 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient(hunting): my global_id is 154226 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f643c003530 con 0x7f6450135b60 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3603085758 0 0) 0x7f64500745e0 con 0x7f645013c220 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.265 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient(hunting): my global_id is 144306 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f645006efd0 con 0x7f645013c220 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3332883525 0 0) 0x7f643c003530 con 0x7f6450135b60 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 >> v1:172.21.15.146:6789/0 conn(0x7f64500a24e0 legacy=0x7f645013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 >> v1:172.21.15.78:6789/0 conn(0x7f645013c220 legacy=0x7f645013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient: _finish_hunting 0 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 monclient: found mon.noname-b 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6450136650 con 0x7f6450135b60 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient: _finish_auth 0 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient: _check_auth_tickets 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:37.266 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 5 monclient: authenticate success, global_id 154226 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f64400016a0 con 0x7f6450135b60 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6454af0640 10 monclient: dump: 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.267 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.253+0000 7f6457d56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f6440001840 con 0x7f6450135b60 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6454af0640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6454af0640 1 -- 172.21.15.78:0/1051781023 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6440001c50 con 0x7f6450135b60 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6454af0640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 4 set_mon_vals no callback set 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6454af0640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6454af0640 10 monclient: dump: 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.268 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: shutdown 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- 172.21.15.78:0/1051781023 >> v1:172.21.15.78:6790/0 conn(0x7f6450135b60 legacy=0x7f6450135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- 172.21.15.78:0/1051781023 shutdown_connections 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- 172.21.15.78:0/1051781023 >> 172.21.15.78:0/1051781023 conn(0x7f64500a63c0 msgr2=0x7f64500a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- 172.21.15.78:0/1051781023 shutdown_connections 2024-09-15T20:50:37.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- 172.21.15.78:0/1051781023 wait complete. 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: build_initial_monmap 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: monmap: 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 librados: starting msgr at 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 librados: starting objecter 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 Processor -- start 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- start start 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 librados: setting wanted keys 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 librados: calling monclient init 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: init 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient(hunting): picked mon.noname-b con 0x7f64500a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient(hunting): picked mon.noname-a con 0x7f6450135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient(hunting): picked mon.noname-c con 0x7f645013c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6450072690 con 0x7f6450135b60 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f645009e2b0 con 0x7f64500a24e0 2024-09-15T20:50:37.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6450072970 con 0x7f645013c220 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient(hunting): authenticate will time out at 2298.146112s 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64552f1640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f6450135b60 0x7f645009dba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46148/0 (socket says 172.21.15.78:46148) 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64552f1640 1 -- 172.21.15.78:0/4165404173 learned_addr learned my addr 172.21.15.78:0/4165404173 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64562f3640 1 --1- 172.21.15.78:0/4165404173 >> v1:172.21.15.78:6790/0 conn(0x7f645013c220 0x7f64501d4c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35376/0 (socket says 172.21.15.78:35376) 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 client.?.objecter ms_handle_connect 0x7f6450135b60 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 client.?.objecter ms_handle_connect 0x7f645013c220 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2994444307 0 0) 0x7f6450072690 con 0x7f6450135b60 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): my global_id is 144309 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6434004ec0 con 0x7f6450135b60 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 615533461 0 0) 0x7f6450072970 con 0x7f645013c220 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): my global_id is 154229 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6450072690 con 0x7f645013c220 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 client.?.objecter ms_handle_connect 0x7f64500a24e0 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2091276791 0 0) 0x7f6434004ec0 con 0x7f6450135b60 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f6450072970 con 0x7f6450135b60 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f644c001a40 con 0x7f6450135b60 2024-09-15T20:50:37.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): dump: 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 648671723 0 0) 0x7f645009e2b0 con 0x7f64500a24e0 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): my global_id is 164125 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6434004ec0 con 0x7f64500a24e0 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3935509130 0 0) 0x7f6450072690 con 0x7f645013c220 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f645009e2b0 con 0x7f645013c220 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f6448001b30 con 0x7f645013c220 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient(hunting): dump: 2024-09-15T20:50:37.275 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2532879068 0 0) 0x7f6450072970 con 0x7f6450135b60 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 >> v1:172.21.15.78:6790/0 conn(0x7f645013c220 legacy=0x7f64501d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 >> v1:172.21.15.146:6789/0 conn(0x7f64500a24e0 legacy=0x7f645009d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient: _finish_hunting 0 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 monclient: found mon.a 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f645013bbb0 con 0x7f6450135b60 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient: _finish_auth 0 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient: _check_auth_tickets 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 5 monclient: authenticate success, global_id 144309 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: _renew_subs 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f645013fa60 con 0x7f6450135b60 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 client.144309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: _renew_subs 2024-09-15T20:50:37.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f645013af70 con 0x7f6450135b60 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 1 librados: init done 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f644c001d40 con 0x7f6450135b60 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f644c003a70 con 0x7f6450135b60 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f64467fc640 10 monclient: dump: 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.278 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6447fff640 4 set_mon_vals no callback set 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.257+0000 7f6457d56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f6457d56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f6457d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64500008e0 con 0x7f6450135b60 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f644c003c10 con 0x7f6450135b60 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f644c04b4f0 con 0x7f6450135b60 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f64467fc640 10 client.144309.objecter ms_dispatch 0x7f6450000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f64467fc640 3 client.144309.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:37.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f64467fc640 3 client.144309.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f64467fc640 20 client.144309.objecter dump_active .. 0 homeless 2024-09-15T20:50:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f6455af2640 10 monclient: get_auth_request con 0x7f643403e1c0 auth_method 0 2024-09-15T20:50:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.261+0000 7f64467fc640 10 client.144309.objecter ms_handle_connect 0x7f643403e1c0 2024-09-15T20:50:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.265+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f644c016bd0 con 0x7f6450135b60 2024-09-15T20:50:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.265+0000 7f64467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.265+0000 7f64467fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:37.501 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.497+0000 7f6457d56640 10 start_mon_command cmd=[{"prefix": "quorum_status"}] 2024-09-15T20:50:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.497+0000 7f6457d56640 10 monclient: _send_command 2 [{"prefix": "quorum_status"}] 2024-09-15T20:50:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.497+0000 7f6457d56640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.497+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f64501357e0 con 0x7f6450135b60 2024-09-15T20:50:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.497+0000 7f64467fc640 1 -- 172.21.15.78:0/4165404173 <== mon.0 v1:172.21.15.78:6789/0 10 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1406 (unknown 254746657 0 1596631538) 0x7f644c0164d0 con 0x7f6450135b60 2024-09-15T20:50:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.497+0000 7f64467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "quorum_status"}] 2024-09-15T20:50:37.502 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.497+0000 7f64467fc640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:50:37.503 INFO:teuthology.orchestra.run.smithi078.stdout:{"election_epoch":80,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540719139924082687","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","modified":"2024-09-15T20:31:01.119418Z","created":"2024-09-15T20:31:01.119418Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6789","nonce":0}]},"addr":"172.21.15.78:6789/0","public_addr":"172.21.15.78:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6789","nonce":0}]},"addr":"172.21.15.146:6789/0","public_addr":"172.21.15.146:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6790","nonce":0}]},"addr":"172.21.15.78:6790/0","public_addr":"172.21.15.78:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 10 librados: watch_flush enter 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 10 librados: watch_flush exit 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 20 client.144309.objecter shutdown clearing up homeless session... 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 10 client.144309.objecter successfully canceled tick 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 >> v1:172.21.15.146:6812/435598642 conn(0x7f643403e1c0 legacy=0x7f6434040680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 10 monclient: shutdown 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 >> v1:172.21.15.78:6789/0 conn(0x7f6450135b60 legacy=0x7f645009dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 shutdown_connections 2024-09-15T20:50:37.506 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 >> 172.21.15.78:0/4165404173 conn(0x7f64500a63c0 msgr2=0x7f645012eaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:37.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 shutdown_connections 2024-09-15T20:50:37.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 1 -- 172.21.15.78:0/4165404173 wait complete. 2024-09-15T20:50:37.507 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.501+0000 7f6457d56640 1 librados: shutdown 2024-09-15T20:50:37.520 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-09-15T20:50:37.521 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-15T20:50:37.530 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-09-15T20:50:37.530 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.645+0000 7f0e5a2ee640 10 monclient: get_monmap_and_config 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.645+0000 7f0e5a2ee640 10 monclient: build_initial_monmap 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.645+0000 7f0e5a2ee640 10 monclient: monmap: 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:37.652075+0000 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:37.652075+0000 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.650 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:37.651 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:37.651 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.658 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.645+0000 7f0e5a2ee640 1 Processor -- start 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- start start 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: init 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): picked mon.noname-a con 0x7f0e540a0ff0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): picked mon.noname-c con 0x7f0e54073800 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): picked mon.noname-b con 0x7f0e540a1ae0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e54072df0 con 0x7f0e540a0ff0 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e54074110 con 0x7f0e54073800 2024-09-15T20:50:37.659 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e540743f0 con 0x7f0e540a1ae0 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): authenticate will time out at 2298.536727s 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e53fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0e540a0ff0 0x7f0e540a13d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46152/0 (socket says 172.21.15.78:46152) 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e53fff640 1 -- 172.21.15.78:0/2824280213 learned_addr learned my addr 172.21.15.78:0/2824280213 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2369958171 0 0) 0x7f0e54072df0 con 0x7f0e540a0ff0 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient(hunting): my global_id is 144312 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e3c003530 con 0x7f0e540a0ff0 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3053109496 0 0) 0x7f0e540743f0 con 0x7f0e540a1ae0 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.660 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient(hunting): my global_id is 154232 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e54072df0 con 0x7f0e540a1ae0 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3307103033 0 0) 0x7f0e3c003530 con 0x7f0e540a0ff0 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 >> v1:172.21.15.78:6790/0 conn(0x7f0e540a1ae0 legacy=0x7f0e54140400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 >> v1:172.21.15.146:6789/0 conn(0x7f0e54073800 legacy=0x7f0e5413ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 monclient: found mon.noname-a 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.661 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e54074eb0 con 0x7f0e540a0ff0 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: _finish_auth 0 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0e440016a0 con 0x7f0e540a0ff0 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: dump: 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.662 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 5 monclient: authenticate success, global_id 144312 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e44001840 con 0x7f0e540a0ff0 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 1 -- 172.21.15.78:0/2824280213 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0e44001c50 con 0x7f0e540a0ff0 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 4 set_mon_vals no callback set 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:37.663 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e52ffd640 10 monclient: dump: 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: shutdown 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/2824280213 >> v1:172.21.15.78:6789/0 conn(0x7f0e540a0ff0 legacy=0x7f0e540a13d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.664 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/2824280213 shutdown_connections 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/2824280213 >> 172.21.15.78:0/2824280213 conn(0x7f0e540a63c0 msgr2=0x7f0e540a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/2824280213 shutdown_connections 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/2824280213 wait complete. 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: build_initial_monmap 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: monmap: 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:37.665 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 librados: starting msgr at 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 librados: starting objecter 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 Processor -- start 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- start start 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 librados: setting wanted keys 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 librados: calling monclient init 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: init 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): picked mon.noname-a con 0x7f0e54073800 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): picked mon.noname-b con 0x7f0e540a0ff0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): picked mon.noname-c con 0x7f0e540a1ae0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:37.666 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e54137580 con 0x7f0e54073800 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e5409dac0 con 0x7f0e540a0ff0 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e5409d180 con 0x7f0e540a1ae0 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e5a2ee640 10 monclient(hunting): authenticate will time out at 2298.539662s 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e53fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0e54073800 0x7f0e54136760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46160/0 (socket says 172.21.15.78:46160) 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e53fff640 1 -- 172.21.15.78:0/1577214131 learned_addr learned my addr 172.21.15.78:0/1577214131 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e50ff9640 10 client.?.objecter ms_handle_connect 0x7f0e54073800 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e50ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e50ff9640 10 client.?.objecter ms_handle_connect 0x7f0e540a1ae0 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e50ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.649+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3552058904 0 0) 0x7f0e54137580 con 0x7f0e54073800 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.667 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): my global_id is 144315 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e40004ec0 con 0x7f0e54073800 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2030680799 0 0) 0x7f0e5409d180 con 0x7f0e540a1ae0 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): my global_id is 154235 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0e54137580 con 0x7f0e540a1ae0 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1692855880 0 0) 0x7f0e40004ec0 con 0x7f0e54073800 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e5409d180 con 0x7f0e54073800 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0e44001b90 con 0x7f0e54073800 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:37.668 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): dump: 2024-09-15T20:50:37.669 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.669 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.669 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.669 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.669 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.669 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.669 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.669 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 client.?.objecter ms_handle_connect 0x7f0e540a0ff0 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 758620029 0 0) 0x7f0e54137580 con 0x7f0e540a1ae0 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0e40004ec0 con 0x7f0e540a1ae0 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0e48001220 con 0x7f0e540a1ae0 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient(hunting): dump: 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.670 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1157741504 0 0) 0x7f0e5409d180 con 0x7f0e54073800 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 >> v1:172.21.15.78:6790/0 conn(0x7f0e540a1ae0 legacy=0x7f0e541d5130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 >> v1:172.21.15.146:6789/0 conn(0x7f0e540a0ff0 legacy=0x7f0e54136e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 monclient: found mon.a 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:37.671 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e5413a550 con 0x7f0e54073800 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient: _finish_auth 0 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 5 monclient: authenticate success, global_id 144315 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 10 monclient: _renew_subs 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0e54139850 con 0x7f0e54073800 2024-09-15T20:50:37.672 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 10 client.144315.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 10 monclient: _renew_subs 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f0e541416a0 con 0x7f0e54073800 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 1 librados: init done 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0e44001820 con 0x7f0e54073800 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0e44004c00 con 0x7f0e54073800 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e527fc640 4 set_mon_vals no callback set 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 monclient: dump: 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.673 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e34005090 con 0x7f0e54073800 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f0e44010c70 con 0x7f0e54073800 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f0e4404b530 con 0x7f0e54073800 2024-09-15T20:50:37.674 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 client.144315.objecter ms_dispatch 0x7f0e54000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 3 client.144315.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 3 client.144315.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 20 client.144315.objecter dump_active .. 0 homeless 2024-09-15T20:50:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e537fe640 10 monclient: get_auth_request con 0x7f0e4003e080 auth_method 0 2024-09-15T20:50:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.653+0000 7f0e50ff9640 10 client.144315.objecter ms_handle_connect 0x7f0e4003e080 2024-09-15T20:50:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.657+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f0e44016ad0 con 0x7f0e54073800 2024-09-15T20:50:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.657+0000 7f0e50ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.675 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.657+0000 7f0e50ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:37.824 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.821+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0e34000ba0 con 0x7f0e4003e080 2024-09-15T20:50:37.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.825+0000 7f0e50ff9640 1 -- 172.21.15.78:0/1577214131 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114126 (unknown 2965378022 0 1130235488) 0x7f0e34000ba0 con 0x7f0e4003e080 2024-09-15T20:50:37.831 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:50:37.832 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:50:37.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 10 librados: watch_flush enter 2024-09-15T20:50:37.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 10 librados: watch_flush exit 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 20 client.144315.objecter shutdown clearing up homeless session... 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 10 client.144315.objecter successfully canceled tick 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 >> v1:172.21.15.146:6812/435598642 conn(0x7f0e4003e080 legacy=0x7f0e40040540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 10 monclient: shutdown 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 >> v1:172.21.15.78:6789/0 conn(0x7f0e54073800 legacy=0x7f0e54136760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 shutdown_connections 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 >> 172.21.15.78:0/1577214131 conn(0x7f0e540a63c0 msgr2=0x7f0e541381d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 shutdown_connections 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 1 -- 172.21.15.78:0/1577214131 wait complete. 2024-09-15T20:50:37.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.829+0000 7f0e5a2ee640 1 librados: shutdown 2024-09-15T20:50:37.847 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":1268,"stamp":"2024-09-15T20:50:37.047922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":409016,"kb_used_data":31672,"kb_used_omap":9,"kb_used_meta":377206,"kb_avail":565822024,"statfs":{"total":579820584960,"available":579401752576,"internally_reserved":0,"allocated":32432128,"data_stored":14246060,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":386259616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003905"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529766+0000","last_change":"2024-09-15T20:32:49.923652+0000","last_active":"2024-09-15T20:49:19.529766+0000","last_peered":"2024-09-15T20:49:19.529766+0000","last_clean":"2024-09-15T20:49:19.529766+0000","last_became_active":"2024-09-15T20:32:49.923378+0000","last_became_peered":"2024-09-15T20:32:49.923378+0000","last_unstale":"2024-09-15T20:49:19.529766+0000","last_undegraded":"2024-09-15T20:49:19.529766+0000","last_fullsized":"2024-09-15T20:49:19.529766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:44.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365025+0000","last_change":"2024-09-15T20:32:49.869206+0000","last_active":"2024-09-15T20:49:20.365025+0000","last_peered":"2024-09-15T20:49:20.365025+0000","last_clean":"2024-09-15T20:49:20.365025+0000","last_became_active":"2024-09-15T20:32:49.868460+0000","last_became_peered":"2024-09-15T20:32:49.868460+0000","last_unstale":"2024-09-15T20:49:20.365025+0000","last_undegraded":"2024-09-15T20:49:20.365025+0000","last_fullsized":"2024-09-15T20:49:20.365025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:52.936657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949298+0000","last_change":"2024-09-15T20:32:49.923995+0000","last_active":"2024-09-15T20:49:19.949298+0000","last_peered":"2024-09-15T20:49:19.949298+0000","last_clean":"2024-09-15T20:49:19.949298+0000","last_became_active":"2024-09-15T20:32:49.923706+0000","last_became_peered":"2024-09-15T20:32:49.923706+0000","last_unstale":"2024-09-15T20:49:19.949298+0000","last_undegraded":"2024-09-15T20:49:19.949298+0000","last_fullsized":"2024-09-15T20:49:19.949298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:23.148031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173672+0000","last_change":"2024-09-15T20:32:49.831383+0000","last_active":"2024-09-15T20:49:18.173672+0000","last_peered":"2024-09-15T20:49:18.173672+0000","last_clean":"2024-09-15T20:49:18.173672+0000","last_became_active":"2024-09-15T20:32:49.830739+0000","last_became_peered":"2024-09-15T20:32:49.830739+0000","last_unstale":"2024-09-15T20:49:18.173672+0000","last_undegraded":"2024-09-15T20:49:18.173672+0000","last_fullsized":"2024-09-15T20:49:18.173672+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:11:38.383945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949357+0000","last_change":"2024-09-15T20:32:49.923640+0000","last_active":"2024-09-15T20:49:19.949357+0000","last_peered":"2024-09-15T20:49:19.949357+0000","last_clean":"2024-09-15T20:49:19.949357+0000","last_became_active":"2024-09-15T20:32:49.921234+0000","last_became_peered":"2024-09-15T20:32:49.921234+0000","last_unstale":"2024-09-15T20:49:19.949357+0000","last_undegraded":"2024-09-15T20:49:19.949357+0000","last_fullsized":"2024-09-15T20:49:19.949357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:20.711668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173971+0000","last_change":"2024-09-15T20:32:49.866040+0000","last_active":"2024-09-15T20:49:18.173971+0000","last_peered":"2024-09-15T20:49:18.173971+0000","last_clean":"2024-09-15T20:49:18.173971+0000","last_became_active":"2024-09-15T20:32:49.864741+0000","last_became_peered":"2024-09-15T20:32:49.864741+0000","last_unstale":"2024-09-15T20:49:18.173971+0000","last_undegraded":"2024-09-15T20:49:18.173971+0000","last_fullsized":"2024-09-15T20:49:18.173971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:00.623365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365414+0000","last_change":"2024-09-15T20:32:49.889906+0000","last_active":"2024-09-15T20:49:20.365414+0000","last_peered":"2024-09-15T20:49:20.365414+0000","last_clean":"2024-09-15T20:49:20.365414+0000","last_became_active":"2024-09-15T20:32:49.887399+0000","last_became_peered":"2024-09-15T20:32:49.887399+0000","last_unstale":"2024-09-15T20:49:20.365414+0000","last_undegraded":"2024-09-15T20:49:20.365414+0000","last_fullsized":"2024-09-15T20:49:20.365414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:02:18.324444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365329+0000","last_change":"2024-09-15T20:32:49.929668+0000","last_active":"2024-09-15T20:49:20.365329+0000","last_peered":"2024-09-15T20:49:20.365329+0000","last_clean":"2024-09-15T20:49:20.365329+0000","last_became_active":"2024-09-15T20:32:49.929513+0000","last_became_peered":"2024-09-15T20:32:49.929513+0000","last_unstale":"2024-09-15T20:49:20.365329+0000","last_undegraded":"2024-09-15T20:49:20.365329+0000","last_fullsized":"2024-09-15T20:49:20.365329+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:53:07.079742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"22'7","reported_seq":1354,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949429+0000","last_change":"2024-09-15T20:32:49.912474+0000","last_active":"2024-09-15T20:49:19.949429+0000","last_peered":"2024-09-15T20:49:19.949429+0000","last_clean":"2024-09-15T20:49:19.949429+0000","last_became_active":"2024-09-15T20:32:49.912168+0000","last_became_peered":"2024-09-15T20:32:49.912168+0000","last_unstale":"2024-09-15T20:49:19.949429+0000","last_undegraded":"2024-09-15T20:49:19.949429+0000","last_fullsized":"2024-09-15T20:49:19.949429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:18.704520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173397+0000","last_change":"2024-09-15T20:32:49.905970+0000","last_active":"2024-09-15T20:49:18.173397+0000","last_peered":"2024-09-15T20:49:18.173397+0000","last_clean":"2024-09-15T20:49:18.173397+0000","last_became_active":"2024-09-15T20:32:49.903676+0000","last_became_peered":"2024-09-15T20:32:49.903676+0000","last_unstale":"2024-09-15T20:49:18.173397+0000","last_undegraded":"2024-09-15T20:49:18.173397+0000","last_fullsized":"2024-09-15T20:49:18.173397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:44:35.092934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317692+0000","last_change":"2024-09-15T20:32:49.832927+0000","last_active":"2024-09-15T20:49:19.317692+0000","last_peered":"2024-09-15T20:49:19.317692+0000","last_clean":"2024-09-15T20:49:19.317692+0000","last_became_active":"2024-09-15T20:32:49.832239+0000","last_became_peered":"2024-09-15T20:32:49.832239+0000","last_unstale":"2024-09-15T20:49:19.317692+0000","last_undegraded":"2024-09-15T20:49:19.317692+0000","last_fullsized":"2024-09-15T20:49:19.317692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:24.870248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173579+0000","last_change":"2024-09-15T20:32:49.963674+0000","last_active":"2024-09-15T20:49:18.173579+0000","last_peered":"2024-09-15T20:49:18.173579+0000","last_clean":"2024-09-15T20:49:18.173579+0000","last_became_active":"2024-09-15T20:32:49.963404+0000","last_became_peered":"2024-09-15T20:32:49.963404+0000","last_unstale":"2024-09-15T20:49:18.173579+0000","last_undegraded":"2024-09-15T20:49:18.173579+0000","last_fullsized":"2024-09-15T20:49:18.173579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:18.112614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1442,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317359+0000","last_change":"2024-09-15T20:31:40.658325+0000","last_active":"2024-09-15T20:49:19.317359+0000","last_peered":"2024-09-15T20:49:19.317359+0000","last_clean":"2024-09-15T20:49:19.317359+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:49:19.317359+0000","last_undegraded":"2024-09-15T20:49:19.317359+0000","last_fullsized":"2024-09-15T20:49:19.317359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_clean_scrub_stamp":"2024-09-15T20:31:38.726961+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:41.743437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365458+0000","last_change":"2024-09-15T20:31:47.833645+0000","last_active":"2024-09-15T20:49:20.365458+0000","last_peered":"2024-09-15T20:49:20.365458+0000","last_clean":"2024-09-15T20:49:20.365458+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:49:20.365458+0000","last_undegraded":"2024-09-15T20:49:20.365458+0000","last_fullsized":"2024-09-15T20:49:20.365458+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:24.343248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1344,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529672+0000","last_change":"2024-09-15T20:32:49.945387+0000","last_active":"2024-09-15T20:49:19.529672+0000","last_peered":"2024-09-15T20:49:19.529672+0000","last_clean":"2024-09-15T20:49:19.529672+0000","last_became_active":"2024-09-15T20:32:49.943293+0000","last_became_peered":"2024-09-15T20:32:49.943293+0000","last_unstale":"2024-09-15T20:49:19.529672+0000","last_undegraded":"2024-09-15T20:49:19.529672+0000","last_fullsized":"2024-09-15T20:49:19.529672+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:58.278471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317229+0000","last_change":"2024-09-15T20:31:47.825565+0000","last_active":"2024-09-15T20:49:19.317229+0000","last_peered":"2024-09-15T20:49:19.317229+0000","last_clean":"2024-09-15T20:49:19.317229+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:49:19.317229+0000","last_undegraded":"2024-09-15T20:49:19.317229+0000","last_fullsized":"2024-09-15T20:49:19.317229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:49:12.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949050+0000","last_change":"2024-09-15T20:32:49.929346+0000","last_active":"2024-09-15T20:49:19.949050+0000","last_peered":"2024-09-15T20:49:19.949050+0000","last_clean":"2024-09-15T20:49:19.949050+0000","last_became_active":"2024-09-15T20:32:49.926970+0000","last_became_peered":"2024-09-15T20:32:49.926970+0000","last_unstale":"2024-09-15T20:49:19.949050+0000","last_undegraded":"2024-09-15T20:49:19.949050+0000","last_fullsized":"2024-09-15T20:49:19.949050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:20.804839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1360,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529794+0000","last_change":"2024-09-15T20:31:47.969706+0000","last_active":"2024-09-15T20:49:19.529794+0000","last_peered":"2024-09-15T20:49:19.529794+0000","last_clean":"2024-09-15T20:49:19.529794+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:49:19.529794+0000","last_undegraded":"2024-09-15T20:49:19.529794+0000","last_fullsized":"2024-09-15T20:49:19.529794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:48.645009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317504+0000","last_change":"2024-09-15T20:32:49.858231+0000","last_active":"2024-09-15T20:49:19.317504+0000","last_peered":"2024-09-15T20:49:19.317504+0000","last_clean":"2024-09-15T20:49:19.317504+0000","last_became_active":"2024-09-15T20:32:49.857975+0000","last_became_peered":"2024-09-15T20:32:49.857975+0000","last_unstale":"2024-09-15T20:49:19.317504+0000","last_undegraded":"2024-09-15T20:49:19.317504+0000","last_fullsized":"2024-09-15T20:49:19.317504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:52.881614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365169+0000","last_change":"2024-09-15T20:31:47.834864+0000","last_active":"2024-09-15T20:49:20.365169+0000","last_peered":"2024-09-15T20:49:20.365169+0000","last_clean":"2024-09-15T20:49:20.365169+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:49:20.365169+0000","last_undegraded":"2024-09-15T20:49:20.365169+0000","last_fullsized":"2024-09-15T20:49:20.365169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:56.096771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949156+0000","last_change":"2024-09-15T20:32:49.950036+0000","last_active":"2024-09-15T20:49:19.949156+0000","last_peered":"2024-09-15T20:49:19.949156+0000","last_clean":"2024-09-15T20:49:19.949156+0000","last_became_active":"2024-09-15T20:32:49.949390+0000","last_became_peered":"2024-09-15T20:32:49.949390+0000","last_unstale":"2024-09-15T20:49:19.949156+0000","last_undegraded":"2024-09-15T20:49:19.949156+0000","last_fullsized":"2024-09-15T20:49:19.949156+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:57.150999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.174094+0000","last_change":"2024-09-15T20:31:47.838237+0000","last_active":"2024-09-15T20:49:18.174094+0000","last_peered":"2024-09-15T20:49:18.174094+0000","last_clean":"2024-09-15T20:49:18.174094+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:49:18.174094+0000","last_undegraded":"2024-09-15T20:49:18.174094+0000","last_fullsized":"2024-09-15T20:49:18.174094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:49.045779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365184+0000","last_change":"2024-09-15T20:32:49.909995+0000","last_active":"2024-09-15T20:49:20.365184+0000","last_peered":"2024-09-15T20:49:20.365184+0000","last_clean":"2024-09-15T20:49:20.365184+0000","last_became_active":"2024-09-15T20:32:49.909430+0000","last_became_peered":"2024-09-15T20:32:49.909430+0000","last_unstale":"2024-09-15T20:49:20.365184+0000","last_undegraded":"2024-09-15T20:49:20.365184+0000","last_fullsized":"2024-09-15T20:49:20.365184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:59:09.239728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317818+0000","last_change":"2024-09-15T20:31:47.825665+0000","last_active":"2024-09-15T20:49:19.317818+0000","last_peered":"2024-09-15T20:49:19.317818+0000","last_clean":"2024-09-15T20:49:19.317818+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:49:19.317818+0000","last_undegraded":"2024-09-15T20:49:19.317818+0000","last_fullsized":"2024-09-15T20:49:19.317818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:40:22.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317704+0000","last_change":"2024-09-15T20:32:49.893856+0000","last_active":"2024-09-15T20:49:19.317704+0000","last_peered":"2024-09-15T20:49:19.317704+0000","last_clean":"2024-09-15T20:49:19.317704+0000","last_became_active":"2024-09-15T20:32:49.893603+0000","last_became_peered":"2024-09-15T20:32:49.893603+0000","last_unstale":"2024-09-15T20:49:19.317704+0000","last_undegraded":"2024-09-15T20:49:19.317704+0000","last_fullsized":"2024-09-15T20:49:19.317704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:18:19.480960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173996+0000","last_change":"2024-09-15T20:31:47.838456+0000","last_active":"2024-09-15T20:49:18.173996+0000","last_peered":"2024-09-15T20:49:18.173996+0000","last_clean":"2024-09-15T20:49:18.173996+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:49:18.173996+0000","last_undegraded":"2024-09-15T20:49:18.173996+0000","last_fullsized":"2024-09-15T20:49:18.173996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:31:54.370290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317442+0000","last_change":"2024-09-15T20:32:49.908877+0000","last_active":"2024-09-15T20:49:19.317442+0000","last_peered":"2024-09-15T20:49:19.317442+0000","last_clean":"2024-09-15T20:49:19.317442+0000","last_became_active":"2024-09-15T20:32:49.908352+0000","last_became_peered":"2024-09-15T20:32:49.908352+0000","last_unstale":"2024-09-15T20:49:19.317442+0000","last_undegraded":"2024-09-15T20:49:19.317442+0000","last_fullsized":"2024-09-15T20:49:19.317442+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:49.714804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317897+0000","last_change":"2024-09-15T20:31:47.826026+0000","last_active":"2024-09-15T20:49:19.317897+0000","last_peered":"2024-09-15T20:49:19.317897+0000","last_clean":"2024-09-15T20:49:19.317897+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:49:19.317897+0000","last_undegraded":"2024-09-15T20:49:19.317897+0000","last_fullsized":"2024-09-15T20:49:19.317897+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:19.315897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529924+0000","last_change":"2024-09-15T20:32:49.973245+0000","last_active":"2024-09-15T20:49:19.529924+0000","last_peered":"2024-09-15T20:49:19.529924+0000","last_clean":"2024-09-15T20:49:19.529924+0000","last_became_active":"2024-09-15T20:32:49.969795+0000","last_became_peered":"2024-09-15T20:32:49.969795+0000","last_unstale":"2024-09-15T20:49:19.529924+0000","last_undegraded":"2024-09-15T20:49:19.529924+0000","last_fullsized":"2024-09-15T20:49:19.529924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:10.478700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949261+0000","last_change":"2024-09-15T20:32:49.948981+0000","last_active":"2024-09-15T20:49:19.949261+0000","last_peered":"2024-09-15T20:49:19.949261+0000","last_clean":"2024-09-15T20:49:19.949261+0000","last_became_active":"2024-09-15T20:32:49.948678+0000","last_became_peered":"2024-09-15T20:32:49.948678+0000","last_unstale":"2024-09-15T20:49:19.949261+0000","last_undegraded":"2024-09-15T20:49:19.949261+0000","last_fullsized":"2024-09-15T20:49:19.949261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:32.657477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.174262+0000","last_change":"2024-09-15T20:32:49.875011+0000","last_active":"2024-09-15T20:49:18.174262+0000","last_peered":"2024-09-15T20:49:18.174262+0000","last_clean":"2024-09-15T20:49:18.174262+0000","last_became_active":"2024-09-15T20:32:49.872991+0000","last_became_peered":"2024-09-15T20:32:49.872991+0000","last_unstale":"2024-09-15T20:49:18.174262+0000","last_undegraded":"2024-09-15T20:49:18.174262+0000","last_fullsized":"2024-09-15T20:49:18.174262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:49.787166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949272+0000","last_change":"2024-09-15T20:32:50.022200+0000","last_active":"2024-09-15T20:49:19.949272+0000","last_peered":"2024-09-15T20:49:19.949272+0000","last_clean":"2024-09-15T20:49:19.949272+0000","last_became_active":"2024-09-15T20:32:50.021808+0000","last_became_peered":"2024-09-15T20:32:50.021808+0000","last_unstale":"2024-09-15T20:49:19.949272+0000","last_undegraded":"2024-09-15T20:49:19.949272+0000","last_fullsized":"2024-09-15T20:49:19.949272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:28:02.730241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949602+0000","last_change":"2024-09-15T20:32:49.946363+0000","last_active":"2024-09-15T20:49:19.949602+0000","last_peered":"2024-09-15T20:49:19.949602+0000","last_clean":"2024-09-15T20:49:19.949602+0000","last_became_active":"2024-09-15T20:32:49.937453+0000","last_became_peered":"2024-09-15T20:32:49.937453+0000","last_unstale":"2024-09-15T20:49:19.949602+0000","last_undegraded":"2024-09-15T20:49:19.949602+0000","last_fullsized":"2024-09-15T20:49:19.949602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:12.866160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.364943+0000","last_change":"2024-09-15T20:32:49.906716+0000","last_active":"2024-09-15T20:49:20.364943+0000","last_peered":"2024-09-15T20:49:20.364943+0000","last_clean":"2024-09-15T20:49:20.364943+0000","last_became_active":"2024-09-15T20:32:49.906485+0000","last_became_peered":"2024-09-15T20:32:49.906485+0000","last_unstale":"2024-09-15T20:49:20.364943+0000","last_undegraded":"2024-09-15T20:49:20.364943+0000","last_fullsized":"2024-09-15T20:49:20.364943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:03:16.163113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317437+0000","last_change":"2024-09-15T20:32:49.846500+0000","last_active":"2024-09-15T20:49:19.317437+0000","last_peered":"2024-09-15T20:49:19.317437+0000","last_clean":"2024-09-15T20:49:19.317437+0000","last_became_active":"2024-09-15T20:32:49.844315+0000","last_became_peered":"2024-09-15T20:32:49.844315+0000","last_unstale":"2024-09-15T20:49:19.317437+0000","last_undegraded":"2024-09-15T20:49:19.317437+0000","last_fullsized":"2024-09-15T20:49:19.317437+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:43:04.920505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327733+0000","last_change":"2024-09-15T20:32:49.935233+0000","last_active":"2024-09-15T20:49:20.327733+0000","last_peered":"2024-09-15T20:49:20.327733+0000","last_clean":"2024-09-15T20:49:20.327733+0000","last_became_active":"2024-09-15T20:32:49.930427+0000","last_became_peered":"2024-09-15T20:32:49.930427+0000","last_unstale":"2024-09-15T20:49:20.327733+0000","last_undegraded":"2024-09-15T20:49:20.327733+0000","last_fullsized":"2024-09-15T20:49:20.327733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:00:31.898076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317298+0000","last_change":"2024-09-15T20:32:49.880288+0000","last_active":"2024-09-15T20:49:19.317298+0000","last_peered":"2024-09-15T20:49:19.317298+0000","last_clean":"2024-09-15T20:49:19.317298+0000","last_became_active":"2024-09-15T20:32:49.880025+0000","last_became_peered":"2024-09-15T20:32:49.880025+0000","last_unstale":"2024-09-15T20:49:19.317298+0000","last_undegraded":"2024-09-15T20:49:19.317298+0000","last_fullsized":"2024-09-15T20:49:19.317298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:36.755630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327939+0000","last_change":"2024-09-15T20:32:49.942013+0000","last_active":"2024-09-15T20:49:20.327939+0000","last_peered":"2024-09-15T20:49:20.327939+0000","last_clean":"2024-09-15T20:49:20.327939+0000","last_became_active":"2024-09-15T20:32:49.939238+0000","last_became_peered":"2024-09-15T20:32:49.939238+0000","last_unstale":"2024-09-15T20:49:20.327939+0000","last_undegraded":"2024-09-15T20:49:20.327939+0000","last_fullsized":"2024-09-15T20:49:20.327939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:23:42.080704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317993+0000","last_change":"2024-09-15T20:32:49.906455+0000","last_active":"2024-09-15T20:49:19.317993+0000","last_peered":"2024-09-15T20:49:19.317993+0000","last_clean":"2024-09-15T20:49:19.317993+0000","last_became_active":"2024-09-15T20:32:49.903971+0000","last_became_peered":"2024-09-15T20:32:49.903971+0000","last_unstale":"2024-09-15T20:49:19.317993+0000","last_undegraded":"2024-09-15T20:49:19.317993+0000","last_fullsized":"2024-09-15T20:49:19.317993+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:03:18.879453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173724+0000","last_change":"2024-09-15T20:32:49.910696+0000","last_active":"2024-09-15T20:49:18.173724+0000","last_peered":"2024-09-15T20:49:18.173724+0000","last_clean":"2024-09-15T20:49:18.173724+0000","last_became_active":"2024-09-15T20:32:49.909018+0000","last_became_peered":"2024-09-15T20:32:49.909018+0000","last_unstale":"2024-09-15T20:49:18.173724+0000","last_undegraded":"2024-09-15T20:49:18.173724+0000","last_fullsized":"2024-09-15T20:49:18.173724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:03.321005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1326,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327891+0000","last_change":"2024-09-15T20:32:49.943386+0000","last_active":"2024-09-15T20:49:20.327891+0000","last_peered":"2024-09-15T20:49:20.327891+0000","last_clean":"2024-09-15T20:49:20.327891+0000","last_became_active":"2024-09-15T20:32:49.942857+0000","last_became_peered":"2024-09-15T20:32:49.942857+0000","last_unstale":"2024-09-15T20:49:20.327891+0000","last_undegraded":"2024-09-15T20:49:20.327891+0000","last_fullsized":"2024-09-15T20:49:20.327891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:01.229283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607782,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72548,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299292,"statfs":{"total":96636764160,"available":96562475008,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5256,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5382144,"data_stored":2198026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607782,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50336,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94321504,"statfs":{"total":96636764160,"available":96585220096,"internally_reserved":0,"allocated":5386240,"data_stored":2198031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607781,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72604,"kb_used_data":5320,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299236,"statfs":{"total":96636764160,"available":96562417664,"internally_reserved":0,"allocated":5447680,"data_stored":2726959,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68440,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303400,"statfs":{"total":96636764160,"available":96566681600,"internally_reserved":0,"allocated":5443584,"data_stored":2726954,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72548,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299292,"statfs":{"total":96636764160,"available":96562475008,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:50:37.850 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: get_monmap_and_config 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: build_initial_monmap 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: monmap: 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:37.969051+0000 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:37.969051+0000 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:37.967 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:37.968 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:37.968 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 Processor -- start 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- start start 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: init 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient(hunting): picked mon.noname-c con 0x7f5f08134bb0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient(hunting): picked mon.noname-a con 0x7f5f0809f5a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0806e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f080a0470 con 0x7f5f0809f5a0 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0809feb0 con 0x7f5f08134bb0 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0809e4a0 con 0x7f5f0806e960 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient(hunting): authenticate will time out at 2298.853765s 2024-09-15T20:50:37.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f077fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5f0809f5a0 0x7f5f081399c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46174/0 (socket says 172.21.15.78:46174) 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f077fe640 1 -- 172.21.15.78:0/1300324223 learned_addr learned my addr 172.21.15.78:0/1300324223 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2437673495 0 0) 0x7f5efc001120 con 0x7f5f0809f5a0 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient(hunting): my global_id is 144318 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef0003530 con 0x7f5f0809f5a0 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 311479007 0 0) 0x7f5f0809e4a0 con 0x7f5f0806e960 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient(hunting): my global_id is 154238 2024-09-15T20:50:37.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5efc001120 con 0x7f5f0806e960 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4034809587 0 0) 0x7f5ef0003530 con 0x7f5f0809f5a0 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 >> v1:172.21.15.78:6790/0 conn(0x7f5f0806e960 legacy=0x7f5f0813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 >> v1:172.21.15.146:6789/0 conn(0x7f5f08134bb0 legacy=0x7f5f08134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 monclient: found mon.noname-a 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f080a0ff0 con 0x7f5f0809f5a0 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: _finish_auth 0 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5efc001870 con 0x7f5f0809f5a0 2024-09-15T20:50:37.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: dump: 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 5 monclient: authenticate success, global_id 144318 2024-09-15T20:50:37.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:37.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5efc001a90 con 0x7f5f0809f5a0 2024-09-15T20:50:37.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:37.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 1 -- 172.21.15.78:0/1300324223 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5efc0048b0 con 0x7f5f0809f5a0 2024-09-15T20:50:37.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:37.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f06ffd640 10 monclient: dump: 2024-09-15T20:50:37.975 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.976 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 4 set_mon_vals no callback set 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: shutdown 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- 172.21.15.78:0/1300324223 >> v1:172.21.15.78:6789/0 conn(0x7f5f0809f5a0 legacy=0x7f5f081399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- 172.21.15.78:0/1300324223 shutdown_connections 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- 172.21.15.78:0/1300324223 >> 172.21.15.78:0/1300324223 conn(0x7f5f080a63c0 msgr2=0x7f5f080a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- 172.21.15.78:0/1300324223 shutdown_connections 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 1 -- 172.21.15.78:0/1300324223 wait complete. 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: build_initial_monmap 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.965+0000 7f5f0e363640 10 monclient: monmap: 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:37.977 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 librados: starting msgr at 2024-09-15T20:50:37.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 librados: starting objecter 2024-09-15T20:50:37.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 Processor -- start 2024-09-15T20:50:37.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 -- start start 2024-09-15T20:50:37.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 librados: setting wanted keys 2024-09-15T20:50:37.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 librados: calling monclient init 2024-09-15T20:50:37.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: init 2024-09-15T20:50:37.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:37.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:37.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient(hunting): picked mon.noname-a con 0x7f5f0806e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:37.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient(hunting): picked mon.noname-b con 0x7f5f0809f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:37.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient(hunting): picked mon.noname-c con 0x7f5f08134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:37.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0811e4a0 con 0x7f5f0806e960 2024-09-15T20:50:37.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0811ff60 con 0x7f5f0809f5a0 2024-09-15T20:50:37.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f0811f000 con 0x7f5f08134bb0 2024-09-15T20:50:37.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:37.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient(hunting): authenticate will time out at 2298.856822s 2024-09-15T20:50:37.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f07fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f5f0806e960 0x7f5f081cc8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46186/0 (socket says 172.21.15.78:46186) 2024-09-15T20:50:37.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f07fff640 1 -- 172.21.15.78:0/1979696486 learned_addr learned my addr 172.21.15.78:0/1979696486 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:37.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 client.?.objecter ms_handle_connect 0x7f5f0806e960 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 client.?.objecter ms_handle_connect 0x7f5f08134bb0 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 146752908 0 0) 0x7f5f0811e4a0 con 0x7f5f0806e960 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): my global_id is 144321 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef4004ef0 con 0x7f5f0806e960 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1703933960 0 0) 0x7f5f0811f000 con 0x7f5f08134bb0 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): my global_id is 154241 2024-09-15T20:50:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5f0811e4a0 con 0x7f5f08134bb0 2024-09-15T20:50:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 client.?.objecter ms_handle_connect 0x7f5f0809f5a0 2024-09-15T20:50:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:37.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3699564130 0 0) 0x7f5ef4004ef0 con 0x7f5f0806e960 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f0811f000 con 0x7f5f0806e960 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5ef8001a40 con 0x7f5f0806e960 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): dump: 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.987 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 309821655 0 0) 0x7f5f0811ff60 con 0x7f5f0809f5a0 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _negotiate 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): my global_id is 164137 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef4004ef0 con 0x7f5f0809f5a0 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1618920022 0 0) 0x7f5f0811e4a0 con 0x7f5f08134bb0 2024-09-15T20:50:37.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5f0811ff60 con 0x7f5f08134bb0 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5f00001c40 con 0x7f5f08134bb0 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient(hunting): dump: 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.989 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2826184408 0 0) 0x7f5f0811f000 con 0x7f5f0806e960 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 >> v1:172.21.15.78:6790/0 conn(0x7f5f08134bb0 legacy=0x7f5f08133550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 >> v1:172.21.15.146:6789/0 conn(0x7f5f0809f5a0 legacy=0x7f5f08132e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient: _finish_hunting 0 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 monclient: found mon.a 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f0811f1d0 con 0x7f5f0806e960 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient: _finish_auth 0 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient: _check_auth_tickets 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 5 monclient: authenticate success, global_id 144321 2024-09-15T20:50:37.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: _renew_subs 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f5f081d5c20 con 0x7f5f0806e960 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 client.144321.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: _renew_subs 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f5f081d5fc0 con 0x7f5f0806e960 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 librados: init done 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5ef8001d40 con 0x7f5f0806e960 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5ef8003a70 con 0x7f5f0806e960 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:37.991 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f067fc640 4 set_mon_vals no callback set 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 monclient: dump: 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.992 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ee8005090 con 0x7f5f0806e960 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f5ef8003c90 con 0x7f5f0806e960 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f5ef804b6f0 con 0x7f5f0806e960 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 client.144321.objecter ms_dispatch 0x7f5f08000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 3 client.144321.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 3 client.144321.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 20 client.144321.objecter dump_active .. 0 homeless 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f077fe640 10 monclient: get_auth_request con 0x7f5ef403e240 auth_method 0 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.969+0000 7f5f04ff9640 10 client.144321.objecter ms_handle_connect 0x7f5ef403e240 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.973+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5f0811f1d0 con 0x7f5f0806e960 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.973+0000 7f5f04ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:37.993 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:37.973+0000 7f5f04ff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:38.144 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.141+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f5ee8000ba0 con 0x7f5ef403e240 2024-09-15T20:50:38.149 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f04ff9640 1 -- 172.21.15.78:0/1979696486 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114126 (unknown 2965378022 0 1130235488) 0x7f5ee8000ba0 con 0x7f5ef403e240 2024-09-15T20:50:38.151 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:50:38.152 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:50:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 10 librados: watch_flush enter 2024-09-15T20:50:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 10 librados: watch_flush exit 2024-09-15T20:50:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 20 client.144321.objecter shutdown clearing up homeless session... 2024-09-15T20:50:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 10 client.144321.objecter successfully canceled tick 2024-09-15T20:50:38.153 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 >> v1:172.21.15.146:6812/435598642 conn(0x7f5ef403e240 legacy=0x7f5ef4040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 10 monclient: shutdown 2024-09-15T20:50:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 >> v1:172.21.15.78:6789/0 conn(0x7f5f0806e960 legacy=0x7f5f081cc8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 shutdown_connections 2024-09-15T20:50:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 >> 172.21.15.78:0/1979696486 conn(0x7f5f080a63c0 msgr2=0x7f5f08133fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 shutdown_connections 2024-09-15T20:50:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 1 -- 172.21.15.78:0/1979696486 wait complete. 2024-09-15T20:50:38.154 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.149+0000 7f5f0e363640 1 librados: shutdown 2024-09-15T20:50:38.169 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":1268,"stamp":"2024-09-15T20:50:37.047922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":409016,"kb_used_data":31672,"kb_used_omap":9,"kb_used_meta":377206,"kb_avail":565822024,"statfs":{"total":579820584960,"available":579401752576,"internally_reserved":0,"allocated":32432128,"data_stored":14246060,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":386259616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003905"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529766+0000","last_change":"2024-09-15T20:32:49.923652+0000","last_active":"2024-09-15T20:49:19.529766+0000","last_peered":"2024-09-15T20:49:19.529766+0000","last_clean":"2024-09-15T20:49:19.529766+0000","last_became_active":"2024-09-15T20:32:49.923378+0000","last_became_peered":"2024-09-15T20:32:49.923378+0000","last_unstale":"2024-09-15T20:49:19.529766+0000","last_undegraded":"2024-09-15T20:49:19.529766+0000","last_fullsized":"2024-09-15T20:49:19.529766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:44.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365025+0000","last_change":"2024-09-15T20:32:49.869206+0000","last_active":"2024-09-15T20:49:20.365025+0000","last_peered":"2024-09-15T20:49:20.365025+0000","last_clean":"2024-09-15T20:49:20.365025+0000","last_became_active":"2024-09-15T20:32:49.868460+0000","last_became_peered":"2024-09-15T20:32:49.868460+0000","last_unstale":"2024-09-15T20:49:20.365025+0000","last_undegraded":"2024-09-15T20:49:20.365025+0000","last_fullsized":"2024-09-15T20:49:20.365025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:52.936657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949298+0000","last_change":"2024-09-15T20:32:49.923995+0000","last_active":"2024-09-15T20:49:19.949298+0000","last_peered":"2024-09-15T20:49:19.949298+0000","last_clean":"2024-09-15T20:49:19.949298+0000","last_became_active":"2024-09-15T20:32:49.923706+0000","last_became_peered":"2024-09-15T20:32:49.923706+0000","last_unstale":"2024-09-15T20:49:19.949298+0000","last_undegraded":"2024-09-15T20:49:19.949298+0000","last_fullsized":"2024-09-15T20:49:19.949298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:23.148031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173672+0000","last_change":"2024-09-15T20:32:49.831383+0000","last_active":"2024-09-15T20:49:18.173672+0000","last_peered":"2024-09-15T20:49:18.173672+0000","last_clean":"2024-09-15T20:49:18.173672+0000","last_became_active":"2024-09-15T20:32:49.830739+0000","last_became_peered":"2024-09-15T20:32:49.830739+0000","last_unstale":"2024-09-15T20:49:18.173672+0000","last_undegraded":"2024-09-15T20:49:18.173672+0000","last_fullsized":"2024-09-15T20:49:18.173672+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:11:38.383945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949357+0000","last_change":"2024-09-15T20:32:49.923640+0000","last_active":"2024-09-15T20:49:19.949357+0000","last_peered":"2024-09-15T20:49:19.949357+0000","last_clean":"2024-09-15T20:49:19.949357+0000","last_became_active":"2024-09-15T20:32:49.921234+0000","last_became_peered":"2024-09-15T20:32:49.921234+0000","last_unstale":"2024-09-15T20:49:19.949357+0000","last_undegraded":"2024-09-15T20:49:19.949357+0000","last_fullsized":"2024-09-15T20:49:19.949357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:20.711668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173971+0000","last_change":"2024-09-15T20:32:49.866040+0000","last_active":"2024-09-15T20:49:18.173971+0000","last_peered":"2024-09-15T20:49:18.173971+0000","last_clean":"2024-09-15T20:49:18.173971+0000","last_became_active":"2024-09-15T20:32:49.864741+0000","last_became_peered":"2024-09-15T20:32:49.864741+0000","last_unstale":"2024-09-15T20:49:18.173971+0000","last_undegraded":"2024-09-15T20:49:18.173971+0000","last_fullsized":"2024-09-15T20:49:18.173971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:00.623365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365414+0000","last_change":"2024-09-15T20:32:49.889906+0000","last_active":"2024-09-15T20:49:20.365414+0000","last_peered":"2024-09-15T20:49:20.365414+0000","last_clean":"2024-09-15T20:49:20.365414+0000","last_became_active":"2024-09-15T20:32:49.887399+0000","last_became_peered":"2024-09-15T20:32:49.887399+0000","last_unstale":"2024-09-15T20:49:20.365414+0000","last_undegraded":"2024-09-15T20:49:20.365414+0000","last_fullsized":"2024-09-15T20:49:20.365414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:02:18.324444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365329+0000","last_change":"2024-09-15T20:32:49.929668+0000","last_active":"2024-09-15T20:49:20.365329+0000","last_peered":"2024-09-15T20:49:20.365329+0000","last_clean":"2024-09-15T20:49:20.365329+0000","last_became_active":"2024-09-15T20:32:49.929513+0000","last_became_peered":"2024-09-15T20:32:49.929513+0000","last_unstale":"2024-09-15T20:49:20.365329+0000","last_undegraded":"2024-09-15T20:49:20.365329+0000","last_fullsized":"2024-09-15T20:49:20.365329+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:53:07.079742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"22'7","reported_seq":1354,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949429+0000","last_change":"2024-09-15T20:32:49.912474+0000","last_active":"2024-09-15T20:49:19.949429+0000","last_peered":"2024-09-15T20:49:19.949429+0000","last_clean":"2024-09-15T20:49:19.949429+0000","last_became_active":"2024-09-15T20:32:49.912168+0000","last_became_peered":"2024-09-15T20:32:49.912168+0000","last_unstale":"2024-09-15T20:49:19.949429+0000","last_undegraded":"2024-09-15T20:49:19.949429+0000","last_fullsized":"2024-09-15T20:49:19.949429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:18.704520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173397+0000","last_change":"2024-09-15T20:32:49.905970+0000","last_active":"2024-09-15T20:49:18.173397+0000","last_peered":"2024-09-15T20:49:18.173397+0000","last_clean":"2024-09-15T20:49:18.173397+0000","last_became_active":"2024-09-15T20:32:49.903676+0000","last_became_peered":"2024-09-15T20:32:49.903676+0000","last_unstale":"2024-09-15T20:49:18.173397+0000","last_undegraded":"2024-09-15T20:49:18.173397+0000","last_fullsized":"2024-09-15T20:49:18.173397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:44:35.092934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317692+0000","last_change":"2024-09-15T20:32:49.832927+0000","last_active":"2024-09-15T20:49:19.317692+0000","last_peered":"2024-09-15T20:49:19.317692+0000","last_clean":"2024-09-15T20:49:19.317692+0000","last_became_active":"2024-09-15T20:32:49.832239+0000","last_became_peered":"2024-09-15T20:32:49.832239+0000","last_unstale":"2024-09-15T20:49:19.317692+0000","last_undegraded":"2024-09-15T20:49:19.317692+0000","last_fullsized":"2024-09-15T20:49:19.317692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:24.870248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173579+0000","last_change":"2024-09-15T20:32:49.963674+0000","last_active":"2024-09-15T20:49:18.173579+0000","last_peered":"2024-09-15T20:49:18.173579+0000","last_clean":"2024-09-15T20:49:18.173579+0000","last_became_active":"2024-09-15T20:32:49.963404+0000","last_became_peered":"2024-09-15T20:32:49.963404+0000","last_unstale":"2024-09-15T20:49:18.173579+0000","last_undegraded":"2024-09-15T20:49:18.173579+0000","last_fullsized":"2024-09-15T20:49:18.173579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:18.112614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1442,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317359+0000","last_change":"2024-09-15T20:31:40.658325+0000","last_active":"2024-09-15T20:49:19.317359+0000","last_peered":"2024-09-15T20:49:19.317359+0000","last_clean":"2024-09-15T20:49:19.317359+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:49:19.317359+0000","last_undegraded":"2024-09-15T20:49:19.317359+0000","last_fullsized":"2024-09-15T20:49:19.317359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_clean_scrub_stamp":"2024-09-15T20:31:38.726961+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:41.743437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365458+0000","last_change":"2024-09-15T20:31:47.833645+0000","last_active":"2024-09-15T20:49:20.365458+0000","last_peered":"2024-09-15T20:49:20.365458+0000","last_clean":"2024-09-15T20:49:20.365458+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:49:20.365458+0000","last_undegraded":"2024-09-15T20:49:20.365458+0000","last_fullsized":"2024-09-15T20:49:20.365458+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:24.343248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1344,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529672+0000","last_change":"2024-09-15T20:32:49.945387+0000","last_active":"2024-09-15T20:49:19.529672+0000","last_peered":"2024-09-15T20:49:19.529672+0000","last_clean":"2024-09-15T20:49:19.529672+0000","last_became_active":"2024-09-15T20:32:49.943293+0000","last_became_peered":"2024-09-15T20:32:49.943293+0000","last_unstale":"2024-09-15T20:49:19.529672+0000","last_undegraded":"2024-09-15T20:49:19.529672+0000","last_fullsized":"2024-09-15T20:49:19.529672+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:58.278471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317229+0000","last_change":"2024-09-15T20:31:47.825565+0000","last_active":"2024-09-15T20:49:19.317229+0000","last_peered":"2024-09-15T20:49:19.317229+0000","last_clean":"2024-09-15T20:49:19.317229+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:49:19.317229+0000","last_undegraded":"2024-09-15T20:49:19.317229+0000","last_fullsized":"2024-09-15T20:49:19.317229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:49:12.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949050+0000","last_change":"2024-09-15T20:32:49.929346+0000","last_active":"2024-09-15T20:49:19.949050+0000","last_peered":"2024-09-15T20:49:19.949050+0000","last_clean":"2024-09-15T20:49:19.949050+0000","last_became_active":"2024-09-15T20:32:49.926970+0000","last_became_peered":"2024-09-15T20:32:49.926970+0000","last_unstale":"2024-09-15T20:49:19.949050+0000","last_undegraded":"2024-09-15T20:49:19.949050+0000","last_fullsized":"2024-09-15T20:49:19.949050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:20.804839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1360,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529794+0000","last_change":"2024-09-15T20:31:47.969706+0000","last_active":"2024-09-15T20:49:19.529794+0000","last_peered":"2024-09-15T20:49:19.529794+0000","last_clean":"2024-09-15T20:49:19.529794+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:49:19.529794+0000","last_undegraded":"2024-09-15T20:49:19.529794+0000","last_fullsized":"2024-09-15T20:49:19.529794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:48.645009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317504+0000","last_change":"2024-09-15T20:32:49.858231+0000","last_active":"2024-09-15T20:49:19.317504+0000","last_peered":"2024-09-15T20:49:19.317504+0000","last_clean":"2024-09-15T20:49:19.317504+0000","last_became_active":"2024-09-15T20:32:49.857975+0000","last_became_peered":"2024-09-15T20:32:49.857975+0000","last_unstale":"2024-09-15T20:49:19.317504+0000","last_undegraded":"2024-09-15T20:49:19.317504+0000","last_fullsized":"2024-09-15T20:49:19.317504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:52.881614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365169+0000","last_change":"2024-09-15T20:31:47.834864+0000","last_active":"2024-09-15T20:49:20.365169+0000","last_peered":"2024-09-15T20:49:20.365169+0000","last_clean":"2024-09-15T20:49:20.365169+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:49:20.365169+0000","last_undegraded":"2024-09-15T20:49:20.365169+0000","last_fullsized":"2024-09-15T20:49:20.365169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:56.096771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949156+0000","last_change":"2024-09-15T20:32:49.950036+0000","last_active":"2024-09-15T20:49:19.949156+0000","last_peered":"2024-09-15T20:49:19.949156+0000","last_clean":"2024-09-15T20:49:19.949156+0000","last_became_active":"2024-09-15T20:32:49.949390+0000","last_became_peered":"2024-09-15T20:32:49.949390+0000","last_unstale":"2024-09-15T20:49:19.949156+0000","last_undegraded":"2024-09-15T20:49:19.949156+0000","last_fullsized":"2024-09-15T20:49:19.949156+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:57.150999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.174094+0000","last_change":"2024-09-15T20:31:47.838237+0000","last_active":"2024-09-15T20:49:18.174094+0000","last_peered":"2024-09-15T20:49:18.174094+0000","last_clean":"2024-09-15T20:49:18.174094+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:49:18.174094+0000","last_undegraded":"2024-09-15T20:49:18.174094+0000","last_fullsized":"2024-09-15T20:49:18.174094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:49.045779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365184+0000","last_change":"2024-09-15T20:32:49.909995+0000","last_active":"2024-09-15T20:49:20.365184+0000","last_peered":"2024-09-15T20:49:20.365184+0000","last_clean":"2024-09-15T20:49:20.365184+0000","last_became_active":"2024-09-15T20:32:49.909430+0000","last_became_peered":"2024-09-15T20:32:49.909430+0000","last_unstale":"2024-09-15T20:49:20.365184+0000","last_undegraded":"2024-09-15T20:49:20.365184+0000","last_fullsized":"2024-09-15T20:49:20.365184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:59:09.239728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317818+0000","last_change":"2024-09-15T20:31:47.825665+0000","last_active":"2024-09-15T20:49:19.317818+0000","last_peered":"2024-09-15T20:49:19.317818+0000","last_clean":"2024-09-15T20:49:19.317818+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:49:19.317818+0000","last_undegraded":"2024-09-15T20:49:19.317818+0000","last_fullsized":"2024-09-15T20:49:19.317818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:40:22.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317704+0000","last_change":"2024-09-15T20:32:49.893856+0000","last_active":"2024-09-15T20:49:19.317704+0000","last_peered":"2024-09-15T20:49:19.317704+0000","last_clean":"2024-09-15T20:49:19.317704+0000","last_became_active":"2024-09-15T20:32:49.893603+0000","last_became_peered":"2024-09-15T20:32:49.893603+0000","last_unstale":"2024-09-15T20:49:19.317704+0000","last_undegraded":"2024-09-15T20:49:19.317704+0000","last_fullsized":"2024-09-15T20:49:19.317704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:18:19.480960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173996+0000","last_change":"2024-09-15T20:31:47.838456+0000","last_active":"2024-09-15T20:49:18.173996+0000","last_peered":"2024-09-15T20:49:18.173996+0000","last_clean":"2024-09-15T20:49:18.173996+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:49:18.173996+0000","last_undegraded":"2024-09-15T20:49:18.173996+0000","last_fullsized":"2024-09-15T20:49:18.173996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:31:54.370290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317442+0000","last_change":"2024-09-15T20:32:49.908877+0000","last_active":"2024-09-15T20:49:19.317442+0000","last_peered":"2024-09-15T20:49:19.317442+0000","last_clean":"2024-09-15T20:49:19.317442+0000","last_became_active":"2024-09-15T20:32:49.908352+0000","last_became_peered":"2024-09-15T20:32:49.908352+0000","last_unstale":"2024-09-15T20:49:19.317442+0000","last_undegraded":"2024-09-15T20:49:19.317442+0000","last_fullsized":"2024-09-15T20:49:19.317442+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:49.714804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317897+0000","last_change":"2024-09-15T20:31:47.826026+0000","last_active":"2024-09-15T20:49:19.317897+0000","last_peered":"2024-09-15T20:49:19.317897+0000","last_clean":"2024-09-15T20:49:19.317897+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:49:19.317897+0000","last_undegraded":"2024-09-15T20:49:19.317897+0000","last_fullsized":"2024-09-15T20:49:19.317897+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:19.315897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529924+0000","last_change":"2024-09-15T20:32:49.973245+0000","last_active":"2024-09-15T20:49:19.529924+0000","last_peered":"2024-09-15T20:49:19.529924+0000","last_clean":"2024-09-15T20:49:19.529924+0000","last_became_active":"2024-09-15T20:32:49.969795+0000","last_became_peered":"2024-09-15T20:32:49.969795+0000","last_unstale":"2024-09-15T20:49:19.529924+0000","last_undegraded":"2024-09-15T20:49:19.529924+0000","last_fullsized":"2024-09-15T20:49:19.529924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:10.478700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949261+0000","last_change":"2024-09-15T20:32:49.948981+0000","last_active":"2024-09-15T20:49:19.949261+0000","last_peered":"2024-09-15T20:49:19.949261+0000","last_clean":"2024-09-15T20:49:19.949261+0000","last_became_active":"2024-09-15T20:32:49.948678+0000","last_became_peered":"2024-09-15T20:32:49.948678+0000","last_unstale":"2024-09-15T20:49:19.949261+0000","last_undegraded":"2024-09-15T20:49:19.949261+0000","last_fullsized":"2024-09-15T20:49:19.949261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:32.657477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.174262+0000","last_change":"2024-09-15T20:32:49.875011+0000","last_active":"2024-09-15T20:49:18.174262+0000","last_peered":"2024-09-15T20:49:18.174262+0000","last_clean":"2024-09-15T20:49:18.174262+0000","last_became_active":"2024-09-15T20:32:49.872991+0000","last_became_peered":"2024-09-15T20:32:49.872991+0000","last_unstale":"2024-09-15T20:49:18.174262+0000","last_undegraded":"2024-09-15T20:49:18.174262+0000","last_fullsized":"2024-09-15T20:49:18.174262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:49.787166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949272+0000","last_change":"2024-09-15T20:32:50.022200+0000","last_active":"2024-09-15T20:49:19.949272+0000","last_peered":"2024-09-15T20:49:19.949272+0000","last_clean":"2024-09-15T20:49:19.949272+0000","last_became_active":"2024-09-15T20:32:50.021808+0000","last_became_peered":"2024-09-15T20:32:50.021808+0000","last_unstale":"2024-09-15T20:49:19.949272+0000","last_undegraded":"2024-09-15T20:49:19.949272+0000","last_fullsized":"2024-09-15T20:49:19.949272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:28:02.730241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949602+0000","last_change":"2024-09-15T20:32:49.946363+0000","last_active":"2024-09-15T20:49:19.949602+0000","last_peered":"2024-09-15T20:49:19.949602+0000","last_clean":"2024-09-15T20:49:19.949602+0000","last_became_active":"2024-09-15T20:32:49.937453+0000","last_became_peered":"2024-09-15T20:32:49.937453+0000","last_unstale":"2024-09-15T20:49:19.949602+0000","last_undegraded":"2024-09-15T20:49:19.949602+0000","last_fullsized":"2024-09-15T20:49:19.949602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:12.866160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.364943+0000","last_change":"2024-09-15T20:32:49.906716+0000","last_active":"2024-09-15T20:49:20.364943+0000","last_peered":"2024-09-15T20:49:20.364943+0000","last_clean":"2024-09-15T20:49:20.364943+0000","last_became_active":"2024-09-15T20:32:49.906485+0000","last_became_peered":"2024-09-15T20:32:49.906485+0000","last_unstale":"2024-09-15T20:49:20.364943+0000","last_undegraded":"2024-09-15T20:49:20.364943+0000","last_fullsized":"2024-09-15T20:49:20.364943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:03:16.163113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317437+0000","last_change":"2024-09-15T20:32:49.846500+0000","last_active":"2024-09-15T20:49:19.317437+0000","last_peered":"2024-09-15T20:49:19.317437+0000","last_clean":"2024-09-15T20:49:19.317437+0000","last_became_active":"2024-09-15T20:32:49.844315+0000","last_became_peered":"2024-09-15T20:32:49.844315+0000","last_unstale":"2024-09-15T20:49:19.317437+0000","last_undegraded":"2024-09-15T20:49:19.317437+0000","last_fullsized":"2024-09-15T20:49:19.317437+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:43:04.920505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327733+0000","last_change":"2024-09-15T20:32:49.935233+0000","last_active":"2024-09-15T20:49:20.327733+0000","last_peered":"2024-09-15T20:49:20.327733+0000","last_clean":"2024-09-15T20:49:20.327733+0000","last_became_active":"2024-09-15T20:32:49.930427+0000","last_became_peered":"2024-09-15T20:32:49.930427+0000","last_unstale":"2024-09-15T20:49:20.327733+0000","last_undegraded":"2024-09-15T20:49:20.327733+0000","last_fullsized":"2024-09-15T20:49:20.327733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:00:31.898076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317298+0000","last_change":"2024-09-15T20:32:49.880288+0000","last_active":"2024-09-15T20:49:19.317298+0000","last_peered":"2024-09-15T20:49:19.317298+0000","last_clean":"2024-09-15T20:49:19.317298+0000","last_became_active":"2024-09-15T20:32:49.880025+0000","last_became_peered":"2024-09-15T20:32:49.880025+0000","last_unstale":"2024-09-15T20:49:19.317298+0000","last_undegraded":"2024-09-15T20:49:19.317298+0000","last_fullsized":"2024-09-15T20:49:19.317298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:36.755630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327939+0000","last_change":"2024-09-15T20:32:49.942013+0000","last_active":"2024-09-15T20:49:20.327939+0000","last_peered":"2024-09-15T20:49:20.327939+0000","last_clean":"2024-09-15T20:49:20.327939+0000","last_became_active":"2024-09-15T20:32:49.939238+0000","last_became_peered":"2024-09-15T20:32:49.939238+0000","last_unstale":"2024-09-15T20:49:20.327939+0000","last_undegraded":"2024-09-15T20:49:20.327939+0000","last_fullsized":"2024-09-15T20:49:20.327939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:23:42.080704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317993+0000","last_change":"2024-09-15T20:32:49.906455+0000","last_active":"2024-09-15T20:49:19.317993+0000","last_peered":"2024-09-15T20:49:19.317993+0000","last_clean":"2024-09-15T20:49:19.317993+0000","last_became_active":"2024-09-15T20:32:49.903971+0000","last_became_peered":"2024-09-15T20:32:49.903971+0000","last_unstale":"2024-09-15T20:49:19.317993+0000","last_undegraded":"2024-09-15T20:49:19.317993+0000","last_fullsized":"2024-09-15T20:49:19.317993+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:03:18.879453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173724+0000","last_change":"2024-09-15T20:32:49.910696+0000","last_active":"2024-09-15T20:49:18.173724+0000","last_peered":"2024-09-15T20:49:18.173724+0000","last_clean":"2024-09-15T20:49:18.173724+0000","last_became_active":"2024-09-15T20:32:49.909018+0000","last_became_peered":"2024-09-15T20:32:49.909018+0000","last_unstale":"2024-09-15T20:49:18.173724+0000","last_undegraded":"2024-09-15T20:49:18.173724+0000","last_fullsized":"2024-09-15T20:49:18.173724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:03.321005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1326,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327891+0000","last_change":"2024-09-15T20:32:49.943386+0000","last_active":"2024-09-15T20:49:20.327891+0000","last_peered":"2024-09-15T20:49:20.327891+0000","last_clean":"2024-09-15T20:49:20.327891+0000","last_became_active":"2024-09-15T20:32:49.942857+0000","last_became_peered":"2024-09-15T20:32:49.942857+0000","last_unstale":"2024-09-15T20:49:20.327891+0000","last_undegraded":"2024-09-15T20:49:20.327891+0000","last_fullsized":"2024-09-15T20:49:20.327891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:01.229283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607782,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72548,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299292,"statfs":{"total":96636764160,"available":96562475008,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5256,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5382144,"data_stored":2198026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607782,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50336,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94321504,"statfs":{"total":96636764160,"available":96585220096,"internally_reserved":0,"allocated":5386240,"data_stored":2198031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607781,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72604,"kb_used_data":5320,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299236,"statfs":{"total":96636764160,"available":96562417664,"internally_reserved":0,"allocated":5447680,"data_stored":2726959,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68440,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303400,"statfs":{"total":96636764160,"available":96566681600,"internally_reserved":0,"allocated":5443584,"data_stored":2726954,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72548,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299292,"statfs":{"total":96636764160,"available":96562475008,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:50:38.171 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-09-15T20:50:38.171 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:50:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.289+0000 7fdcb107c640 10 monclient: get_monmap_and_config 2024-09-15T20:50:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.289+0000 7fdcb107c640 10 monclient: build_initial_monmap 2024-09-15T20:50:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.289+0000 7fdcb107c640 10 monclient: monmap: 2024-09-15T20:50:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:38.296312+0000 2024-09-15T20:50:38.294 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:38.296312+0000 2024-09-15T20:50:38.295 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:38.295 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.295 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:38.295 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:38.295 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:38.295 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 Processor -- start 2024-09-15T20:50:38.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- start start 2024-09-15T20:50:38.303 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: init 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): picked mon.noname-b con 0x7fdcac137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): picked mon.noname-c con 0x7fdcac0a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): picked mon.noname-a con 0x7fdcac13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcac0a6420 con 0x7fdcac13e5f0 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcac0a6700 con 0x7fdcac0a9610 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcac0a2110 con 0x7fdcac137ce0 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): authenticate will time out at 2299.180821s 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcab59e640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdcac13e5f0 0x7fdcac1409e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46192/0 (socket says 172.21.15.78:46192) 2024-09-15T20:50:38.304 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcab59e640 1 -- 172.21.15.78:0/953978614 learned_addr learned my addr 172.21.15.78:0/953978614 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3496041309 0 0) 0x7fdcac0a6420 con 0x7fdcac13e5f0 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient(hunting): my global_id is 144324 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc94003530 con 0x7fdcac13e5f0 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2683501751 0 0) 0x7fdcac0a2110 con 0x7fdcac137ce0 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient(hunting): my global_id is 154244 2024-09-15T20:50:38.305 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcac0a6420 con 0x7fdcac137ce0 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3576992738 0 0) 0x7fdc94003530 con 0x7fdcac13e5f0 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 >> v1:172.21.15.78:6790/0 conn(0x7fdcac137ce0 legacy=0x7fdcac1380c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 >> v1:172.21.15.146:6789/0 conn(0x7fdcac0a9610 legacy=0x7fdcac13ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: _finish_hunting 0 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 monclient: found mon.noname-a 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac0a4bc0 con 0x7fdcac13e5f0 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: _finish_auth 0 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: _check_auth_tickets 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:38.306 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdca4001a20 con 0x7fdcac13e5f0 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: dump: 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.307 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 5 monclient: authenticate success, global_id 144324 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdca4001bc0 con 0x7fdcac13e5f0 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 1 -- 172.21.15.78:0/953978614 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdca4001fd0 con 0x7fdcac13e5f0 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 4 set_mon_vals no callback set 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdca9d9b640 10 monclient: dump: 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.308 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: shutdown 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- 172.21.15.78:0/953978614 >> v1:172.21.15.78:6789/0 conn(0x7fdcac13e5f0 legacy=0x7fdcac1409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- 172.21.15.78:0/953978614 shutdown_connections 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- 172.21.15.78:0/953978614 >> 172.21.15.78:0/953978614 conn(0x7fdcac0aa6c0 msgr2=0x7fdcac0aaad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- 172.21.15.78:0/953978614 shutdown_connections 2024-09-15T20:50:38.309 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- 172.21.15.78:0/953978614 wait complete. 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: build_initial_monmap 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: monmap: 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 librados: starting msgr at 2024-09-15T20:50:38.310 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 librados: starting objecter 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 Processor -- start 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- start start 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 librados: setting wanted keys 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 librados: calling monclient init 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: init 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): picked mon.noname-b con 0x7fdcac0a9610 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): picked mon.noname-c con 0x7fdcac137ce0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): picked mon.noname-a con 0x7fdcac13e5f0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcac13c010 con 0x7fdcac13e5f0 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcac13c2f0 con 0x7fdcac0a9610 2024-09-15T20:50:38.311 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcac0a14b0 con 0x7fdcac137ce0 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcb107c640 10 monclient(hunting): authenticate will time out at 2299.183932s 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcaa59c640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fdcac137ce0 0x7fdcac1d1870 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35452/0 (socket says 172.21.15.78:35452) 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.293+0000 7fdcaa59c640 1 -- 172.21.15.78:0/1475284111 learned_addr learned my addr 172.21.15.78:0/1475284111 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 client.?.objecter ms_handle_connect 0x7fdcac137ce0 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 client.?.objecter ms_handle_connect 0x7fdcac13e5f0 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 23415320 0 0) 0x7fdcac0a14b0 con 0x7fdcac137ce0 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.312 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): my global_id is 154247 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdc98004ec0 con 0x7fdcac137ce0 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1523740777 0 0) 0x7fdcac13c010 con 0x7fdcac13e5f0 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): my global_id is 144327 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcac0a14b0 con 0x7fdcac13e5f0 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 client.?.objecter ms_handle_connect 0x7fdcac0a9610 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4053750297 0 0) 0x7fdc98004ec0 con 0x7fdcac137ce0 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdcac13c010 con 0x7fdcac137ce0 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdca0001a40 con 0x7fdcac137ce0 2024-09-15T20:50:38.313 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): dump: 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:38.314 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 888265311 0 0) 0x7fdcac0a14b0 con 0x7fdcac13e5f0 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdc98004ec0 con 0x7fdcac13e5f0 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdca4001d70 con 0x7fdcac13e5f0 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): dump: 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.315 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3279082372 0 0) 0x7fdcac13c2f0 con 0x7fdcac0a9610 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient(hunting): my global_id is 164143 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdcac0a14b0 con 0x7fdcac0a9610 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2234815604 0 0) 0x7fdcac13c010 con 0x7fdcac137ce0 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 >> v1:172.21.15.146:6789/0 conn(0x7fdcac0a9610 legacy=0x7fdcac1cca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 >> v1:172.21.15.78:6789/0 conn(0x7fdcac13e5f0 legacy=0x7fdcac1d4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.316 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient: _finish_hunting 0 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 monclient: found mon.c 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcac139f90 con 0x7fdcac137ce0 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient: _finish_auth 0 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient: _check_auth_tickets 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 5 monclient: authenticate success, global_id 154247 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 10 monclient: _renew_subs 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdcac13ae30 con 0x7fdcac137ce0 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 10 client.154247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:38.317 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 10 monclient: _renew_subs 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fdcac1cd120 con 0x7fdcac137ce0 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 1 librados: init done 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdca0001cc0 con 0x7fdcac137ce0 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdca0003a70 con 0x7fdcac137ce0 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:38.318 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdca959a640 4 set_mon_vals no callback set 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 monclient: dump: 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:38.319 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc8c005090 con 0x7fdcac137ce0 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fdca0003c90 con 0x7fdcac137ce0 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fdca004b600 con 0x7fdcac137ce0 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 client.154247.objecter ms_dispatch 0x7fdcac000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 3 client.154247.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 3 client.154247.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 20 client.154247.objecter dump_active .. 0 homeless 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdcaad9d640 10 monclient: get_auth_request con 0x7fdc9803e2b0 auth_method 0 2024-09-15T20:50:38.320 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.297+0000 7fdc937fe640 10 client.154247.objecter ms_handle_connect 0x7fdc9803e2b0 2024-09-15T20:50:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.301+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fdca0016b60 con 0x7fdcac137ce0 2024-09-15T20:50:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.301+0000 7fdc937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:38.321 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.301+0000 7fdc937fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:38.471 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.469+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdc8c000ba0 con 0x7fdc9803e2b0 2024-09-15T20:50:38.477 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.473+0000 7fdc937fe640 1 -- 172.21.15.78:0/1475284111 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114126 (unknown 2965378022 0 1130235488) 0x7fdc8c000ba0 con 0x7fdc9803e2b0 2024-09-15T20:50:38.478 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:50:38.479 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:50:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 10 librados: watch_flush enter 2024-09-15T20:50:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 10 librados: watch_flush exit 2024-09-15T20:50:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 20 client.154247.objecter shutdown clearing up homeless session... 2024-09-15T20:50:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 10 client.154247.objecter successfully canceled tick 2024-09-15T20:50:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 >> v1:172.21.15.146:6812/435598642 conn(0x7fdc9803e2b0 legacy=0x7fdc98040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 10 monclient: shutdown 2024-09-15T20:50:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:38.480 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 >> v1:172.21.15.78:6790/0 conn(0x7fdcac137ce0 legacy=0x7fdcac1d1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 shutdown_connections 2024-09-15T20:50:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 >> 172.21.15.78:0/1475284111 conn(0x7fdcac0aa6c0 msgr2=0x7fdcac1409e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 shutdown_connections 2024-09-15T20:50:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 1 -- 172.21.15.78:0/1475284111 wait complete. 2024-09-15T20:50:38.481 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.477+0000 7fdcb107c640 1 librados: shutdown 2024-09-15T20:50:38.495 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":1268,"stamp":"2024-09-15T20:50:37.047922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":409016,"kb_used_data":31672,"kb_used_omap":9,"kb_used_meta":377206,"kb_avail":565822024,"statfs":{"total":579820584960,"available":579401752576,"internally_reserved":0,"allocated":32432128,"data_stored":14246060,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":386259616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003905"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529766+0000","last_change":"2024-09-15T20:32:49.923652+0000","last_active":"2024-09-15T20:49:19.529766+0000","last_peered":"2024-09-15T20:49:19.529766+0000","last_clean":"2024-09-15T20:49:19.529766+0000","last_became_active":"2024-09-15T20:32:49.923378+0000","last_became_peered":"2024-09-15T20:32:49.923378+0000","last_unstale":"2024-09-15T20:49:19.529766+0000","last_undegraded":"2024-09-15T20:49:19.529766+0000","last_fullsized":"2024-09-15T20:49:19.529766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:44.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365025+0000","last_change":"2024-09-15T20:32:49.869206+0000","last_active":"2024-09-15T20:49:20.365025+0000","last_peered":"2024-09-15T20:49:20.365025+0000","last_clean":"2024-09-15T20:49:20.365025+0000","last_became_active":"2024-09-15T20:32:49.868460+0000","last_became_peered":"2024-09-15T20:32:49.868460+0000","last_unstale":"2024-09-15T20:49:20.365025+0000","last_undegraded":"2024-09-15T20:49:20.365025+0000","last_fullsized":"2024-09-15T20:49:20.365025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:52.936657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949298+0000","last_change":"2024-09-15T20:32:49.923995+0000","last_active":"2024-09-15T20:49:19.949298+0000","last_peered":"2024-09-15T20:49:19.949298+0000","last_clean":"2024-09-15T20:49:19.949298+0000","last_became_active":"2024-09-15T20:32:49.923706+0000","last_became_peered":"2024-09-15T20:32:49.923706+0000","last_unstale":"2024-09-15T20:49:19.949298+0000","last_undegraded":"2024-09-15T20:49:19.949298+0000","last_fullsized":"2024-09-15T20:49:19.949298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:23.148031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173672+0000","last_change":"2024-09-15T20:32:49.831383+0000","last_active":"2024-09-15T20:49:18.173672+0000","last_peered":"2024-09-15T20:49:18.173672+0000","last_clean":"2024-09-15T20:49:18.173672+0000","last_became_active":"2024-09-15T20:32:49.830739+0000","last_became_peered":"2024-09-15T20:32:49.830739+0000","last_unstale":"2024-09-15T20:49:18.173672+0000","last_undegraded":"2024-09-15T20:49:18.173672+0000","last_fullsized":"2024-09-15T20:49:18.173672+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:11:38.383945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949357+0000","last_change":"2024-09-15T20:32:49.923640+0000","last_active":"2024-09-15T20:49:19.949357+0000","last_peered":"2024-09-15T20:49:19.949357+0000","last_clean":"2024-09-15T20:49:19.949357+0000","last_became_active":"2024-09-15T20:32:49.921234+0000","last_became_peered":"2024-09-15T20:32:49.921234+0000","last_unstale":"2024-09-15T20:49:19.949357+0000","last_undegraded":"2024-09-15T20:49:19.949357+0000","last_fullsized":"2024-09-15T20:49:19.949357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:20.711668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173971+0000","last_change":"2024-09-15T20:32:49.866040+0000","last_active":"2024-09-15T20:49:18.173971+0000","last_peered":"2024-09-15T20:49:18.173971+0000","last_clean":"2024-09-15T20:49:18.173971+0000","last_became_active":"2024-09-15T20:32:49.864741+0000","last_became_peered":"2024-09-15T20:32:49.864741+0000","last_unstale":"2024-09-15T20:49:18.173971+0000","last_undegraded":"2024-09-15T20:49:18.173971+0000","last_fullsized":"2024-09-15T20:49:18.173971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:00.623365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365414+0000","last_change":"2024-09-15T20:32:49.889906+0000","last_active":"2024-09-15T20:49:20.365414+0000","last_peered":"2024-09-15T20:49:20.365414+0000","last_clean":"2024-09-15T20:49:20.365414+0000","last_became_active":"2024-09-15T20:32:49.887399+0000","last_became_peered":"2024-09-15T20:32:49.887399+0000","last_unstale":"2024-09-15T20:49:20.365414+0000","last_undegraded":"2024-09-15T20:49:20.365414+0000","last_fullsized":"2024-09-15T20:49:20.365414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:02:18.324444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365329+0000","last_change":"2024-09-15T20:32:49.929668+0000","last_active":"2024-09-15T20:49:20.365329+0000","last_peered":"2024-09-15T20:49:20.365329+0000","last_clean":"2024-09-15T20:49:20.365329+0000","last_became_active":"2024-09-15T20:32:49.929513+0000","last_became_peered":"2024-09-15T20:32:49.929513+0000","last_unstale":"2024-09-15T20:49:20.365329+0000","last_undegraded":"2024-09-15T20:49:20.365329+0000","last_fullsized":"2024-09-15T20:49:20.365329+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:53:07.079742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"22'7","reported_seq":1354,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949429+0000","last_change":"2024-09-15T20:32:49.912474+0000","last_active":"2024-09-15T20:49:19.949429+0000","last_peered":"2024-09-15T20:49:19.949429+0000","last_clean":"2024-09-15T20:49:19.949429+0000","last_became_active":"2024-09-15T20:32:49.912168+0000","last_became_peered":"2024-09-15T20:32:49.912168+0000","last_unstale":"2024-09-15T20:49:19.949429+0000","last_undegraded":"2024-09-15T20:49:19.949429+0000","last_fullsized":"2024-09-15T20:49:19.949429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:18.704520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173397+0000","last_change":"2024-09-15T20:32:49.905970+0000","last_active":"2024-09-15T20:49:18.173397+0000","last_peered":"2024-09-15T20:49:18.173397+0000","last_clean":"2024-09-15T20:49:18.173397+0000","last_became_active":"2024-09-15T20:32:49.903676+0000","last_became_peered":"2024-09-15T20:32:49.903676+0000","last_unstale":"2024-09-15T20:49:18.173397+0000","last_undegraded":"2024-09-15T20:49:18.173397+0000","last_fullsized":"2024-09-15T20:49:18.173397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:44:35.092934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317692+0000","last_change":"2024-09-15T20:32:49.832927+0000","last_active":"2024-09-15T20:49:19.317692+0000","last_peered":"2024-09-15T20:49:19.317692+0000","last_clean":"2024-09-15T20:49:19.317692+0000","last_became_active":"2024-09-15T20:32:49.832239+0000","last_became_peered":"2024-09-15T20:32:49.832239+0000","last_unstale":"2024-09-15T20:49:19.317692+0000","last_undegraded":"2024-09-15T20:49:19.317692+0000","last_fullsized":"2024-09-15T20:49:19.317692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:24.870248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173579+0000","last_change":"2024-09-15T20:32:49.963674+0000","last_active":"2024-09-15T20:49:18.173579+0000","last_peered":"2024-09-15T20:49:18.173579+0000","last_clean":"2024-09-15T20:49:18.173579+0000","last_became_active":"2024-09-15T20:32:49.963404+0000","last_became_peered":"2024-09-15T20:32:49.963404+0000","last_unstale":"2024-09-15T20:49:18.173579+0000","last_undegraded":"2024-09-15T20:49:18.173579+0000","last_fullsized":"2024-09-15T20:49:18.173579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:18.112614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1442,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317359+0000","last_change":"2024-09-15T20:31:40.658325+0000","last_active":"2024-09-15T20:49:19.317359+0000","last_peered":"2024-09-15T20:49:19.317359+0000","last_clean":"2024-09-15T20:49:19.317359+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:49:19.317359+0000","last_undegraded":"2024-09-15T20:49:19.317359+0000","last_fullsized":"2024-09-15T20:49:19.317359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_clean_scrub_stamp":"2024-09-15T20:31:38.726961+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:41.743437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365458+0000","last_change":"2024-09-15T20:31:47.833645+0000","last_active":"2024-09-15T20:49:20.365458+0000","last_peered":"2024-09-15T20:49:20.365458+0000","last_clean":"2024-09-15T20:49:20.365458+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:49:20.365458+0000","last_undegraded":"2024-09-15T20:49:20.365458+0000","last_fullsized":"2024-09-15T20:49:20.365458+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:24.343248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1344,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529672+0000","last_change":"2024-09-15T20:32:49.945387+0000","last_active":"2024-09-15T20:49:19.529672+0000","last_peered":"2024-09-15T20:49:19.529672+0000","last_clean":"2024-09-15T20:49:19.529672+0000","last_became_active":"2024-09-15T20:32:49.943293+0000","last_became_peered":"2024-09-15T20:32:49.943293+0000","last_unstale":"2024-09-15T20:49:19.529672+0000","last_undegraded":"2024-09-15T20:49:19.529672+0000","last_fullsized":"2024-09-15T20:49:19.529672+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:58.278471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317229+0000","last_change":"2024-09-15T20:31:47.825565+0000","last_active":"2024-09-15T20:49:19.317229+0000","last_peered":"2024-09-15T20:49:19.317229+0000","last_clean":"2024-09-15T20:49:19.317229+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:49:19.317229+0000","last_undegraded":"2024-09-15T20:49:19.317229+0000","last_fullsized":"2024-09-15T20:49:19.317229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:49:12.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949050+0000","last_change":"2024-09-15T20:32:49.929346+0000","last_active":"2024-09-15T20:49:19.949050+0000","last_peered":"2024-09-15T20:49:19.949050+0000","last_clean":"2024-09-15T20:49:19.949050+0000","last_became_active":"2024-09-15T20:32:49.926970+0000","last_became_peered":"2024-09-15T20:32:49.926970+0000","last_unstale":"2024-09-15T20:49:19.949050+0000","last_undegraded":"2024-09-15T20:49:19.949050+0000","last_fullsized":"2024-09-15T20:49:19.949050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:20.804839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1360,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529794+0000","last_change":"2024-09-15T20:31:47.969706+0000","last_active":"2024-09-15T20:49:19.529794+0000","last_peered":"2024-09-15T20:49:19.529794+0000","last_clean":"2024-09-15T20:49:19.529794+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:49:19.529794+0000","last_undegraded":"2024-09-15T20:49:19.529794+0000","last_fullsized":"2024-09-15T20:49:19.529794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:48.645009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317504+0000","last_change":"2024-09-15T20:32:49.858231+0000","last_active":"2024-09-15T20:49:19.317504+0000","last_peered":"2024-09-15T20:49:19.317504+0000","last_clean":"2024-09-15T20:49:19.317504+0000","last_became_active":"2024-09-15T20:32:49.857975+0000","last_became_peered":"2024-09-15T20:32:49.857975+0000","last_unstale":"2024-09-15T20:49:19.317504+0000","last_undegraded":"2024-09-15T20:49:19.317504+0000","last_fullsized":"2024-09-15T20:49:19.317504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:52.881614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365169+0000","last_change":"2024-09-15T20:31:47.834864+0000","last_active":"2024-09-15T20:49:20.365169+0000","last_peered":"2024-09-15T20:49:20.365169+0000","last_clean":"2024-09-15T20:49:20.365169+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:49:20.365169+0000","last_undegraded":"2024-09-15T20:49:20.365169+0000","last_fullsized":"2024-09-15T20:49:20.365169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:56.096771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949156+0000","last_change":"2024-09-15T20:32:49.950036+0000","last_active":"2024-09-15T20:49:19.949156+0000","last_peered":"2024-09-15T20:49:19.949156+0000","last_clean":"2024-09-15T20:49:19.949156+0000","last_became_active":"2024-09-15T20:32:49.949390+0000","last_became_peered":"2024-09-15T20:32:49.949390+0000","last_unstale":"2024-09-15T20:49:19.949156+0000","last_undegraded":"2024-09-15T20:49:19.949156+0000","last_fullsized":"2024-09-15T20:49:19.949156+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:57.150999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.174094+0000","last_change":"2024-09-15T20:31:47.838237+0000","last_active":"2024-09-15T20:49:18.174094+0000","last_peered":"2024-09-15T20:49:18.174094+0000","last_clean":"2024-09-15T20:49:18.174094+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:49:18.174094+0000","last_undegraded":"2024-09-15T20:49:18.174094+0000","last_fullsized":"2024-09-15T20:49:18.174094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:49.045779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365184+0000","last_change":"2024-09-15T20:32:49.909995+0000","last_active":"2024-09-15T20:49:20.365184+0000","last_peered":"2024-09-15T20:49:20.365184+0000","last_clean":"2024-09-15T20:49:20.365184+0000","last_became_active":"2024-09-15T20:32:49.909430+0000","last_became_peered":"2024-09-15T20:32:49.909430+0000","last_unstale":"2024-09-15T20:49:20.365184+0000","last_undegraded":"2024-09-15T20:49:20.365184+0000","last_fullsized":"2024-09-15T20:49:20.365184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:59:09.239728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317818+0000","last_change":"2024-09-15T20:31:47.825665+0000","last_active":"2024-09-15T20:49:19.317818+0000","last_peered":"2024-09-15T20:49:19.317818+0000","last_clean":"2024-09-15T20:49:19.317818+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:49:19.317818+0000","last_undegraded":"2024-09-15T20:49:19.317818+0000","last_fullsized":"2024-09-15T20:49:19.317818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:40:22.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317704+0000","last_change":"2024-09-15T20:32:49.893856+0000","last_active":"2024-09-15T20:49:19.317704+0000","last_peered":"2024-09-15T20:49:19.317704+0000","last_clean":"2024-09-15T20:49:19.317704+0000","last_became_active":"2024-09-15T20:32:49.893603+0000","last_became_peered":"2024-09-15T20:32:49.893603+0000","last_unstale":"2024-09-15T20:49:19.317704+0000","last_undegraded":"2024-09-15T20:49:19.317704+0000","last_fullsized":"2024-09-15T20:49:19.317704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:18:19.480960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173996+0000","last_change":"2024-09-15T20:31:47.838456+0000","last_active":"2024-09-15T20:49:18.173996+0000","last_peered":"2024-09-15T20:49:18.173996+0000","last_clean":"2024-09-15T20:49:18.173996+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:49:18.173996+0000","last_undegraded":"2024-09-15T20:49:18.173996+0000","last_fullsized":"2024-09-15T20:49:18.173996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:31:54.370290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317442+0000","last_change":"2024-09-15T20:32:49.908877+0000","last_active":"2024-09-15T20:49:19.317442+0000","last_peered":"2024-09-15T20:49:19.317442+0000","last_clean":"2024-09-15T20:49:19.317442+0000","last_became_active":"2024-09-15T20:32:49.908352+0000","last_became_peered":"2024-09-15T20:32:49.908352+0000","last_unstale":"2024-09-15T20:49:19.317442+0000","last_undegraded":"2024-09-15T20:49:19.317442+0000","last_fullsized":"2024-09-15T20:49:19.317442+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:49.714804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317897+0000","last_change":"2024-09-15T20:31:47.826026+0000","last_active":"2024-09-15T20:49:19.317897+0000","last_peered":"2024-09-15T20:49:19.317897+0000","last_clean":"2024-09-15T20:49:19.317897+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:49:19.317897+0000","last_undegraded":"2024-09-15T20:49:19.317897+0000","last_fullsized":"2024-09-15T20:49:19.317897+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:19.315897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529924+0000","last_change":"2024-09-15T20:32:49.973245+0000","last_active":"2024-09-15T20:49:19.529924+0000","last_peered":"2024-09-15T20:49:19.529924+0000","last_clean":"2024-09-15T20:49:19.529924+0000","last_became_active":"2024-09-15T20:32:49.969795+0000","last_became_peered":"2024-09-15T20:32:49.969795+0000","last_unstale":"2024-09-15T20:49:19.529924+0000","last_undegraded":"2024-09-15T20:49:19.529924+0000","last_fullsized":"2024-09-15T20:49:19.529924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:10.478700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949261+0000","last_change":"2024-09-15T20:32:49.948981+0000","last_active":"2024-09-15T20:49:19.949261+0000","last_peered":"2024-09-15T20:49:19.949261+0000","last_clean":"2024-09-15T20:49:19.949261+0000","last_became_active":"2024-09-15T20:32:49.948678+0000","last_became_peered":"2024-09-15T20:32:49.948678+0000","last_unstale":"2024-09-15T20:49:19.949261+0000","last_undegraded":"2024-09-15T20:49:19.949261+0000","last_fullsized":"2024-09-15T20:49:19.949261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:32.657477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.174262+0000","last_change":"2024-09-15T20:32:49.875011+0000","last_active":"2024-09-15T20:49:18.174262+0000","last_peered":"2024-09-15T20:49:18.174262+0000","last_clean":"2024-09-15T20:49:18.174262+0000","last_became_active":"2024-09-15T20:32:49.872991+0000","last_became_peered":"2024-09-15T20:32:49.872991+0000","last_unstale":"2024-09-15T20:49:18.174262+0000","last_undegraded":"2024-09-15T20:49:18.174262+0000","last_fullsized":"2024-09-15T20:49:18.174262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:49.787166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949272+0000","last_change":"2024-09-15T20:32:50.022200+0000","last_active":"2024-09-15T20:49:19.949272+0000","last_peered":"2024-09-15T20:49:19.949272+0000","last_clean":"2024-09-15T20:49:19.949272+0000","last_became_active":"2024-09-15T20:32:50.021808+0000","last_became_peered":"2024-09-15T20:32:50.021808+0000","last_unstale":"2024-09-15T20:49:19.949272+0000","last_undegraded":"2024-09-15T20:49:19.949272+0000","last_fullsized":"2024-09-15T20:49:19.949272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:28:02.730241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949602+0000","last_change":"2024-09-15T20:32:49.946363+0000","last_active":"2024-09-15T20:49:19.949602+0000","last_peered":"2024-09-15T20:49:19.949602+0000","last_clean":"2024-09-15T20:49:19.949602+0000","last_became_active":"2024-09-15T20:32:49.937453+0000","last_became_peered":"2024-09-15T20:32:49.937453+0000","last_unstale":"2024-09-15T20:49:19.949602+0000","last_undegraded":"2024-09-15T20:49:19.949602+0000","last_fullsized":"2024-09-15T20:49:19.949602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:12.866160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.364943+0000","last_change":"2024-09-15T20:32:49.906716+0000","last_active":"2024-09-15T20:49:20.364943+0000","last_peered":"2024-09-15T20:49:20.364943+0000","last_clean":"2024-09-15T20:49:20.364943+0000","last_became_active":"2024-09-15T20:32:49.906485+0000","last_became_peered":"2024-09-15T20:32:49.906485+0000","last_unstale":"2024-09-15T20:49:20.364943+0000","last_undegraded":"2024-09-15T20:49:20.364943+0000","last_fullsized":"2024-09-15T20:49:20.364943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:03:16.163113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317437+0000","last_change":"2024-09-15T20:32:49.846500+0000","last_active":"2024-09-15T20:49:19.317437+0000","last_peered":"2024-09-15T20:49:19.317437+0000","last_clean":"2024-09-15T20:49:19.317437+0000","last_became_active":"2024-09-15T20:32:49.844315+0000","last_became_peered":"2024-09-15T20:32:49.844315+0000","last_unstale":"2024-09-15T20:49:19.317437+0000","last_undegraded":"2024-09-15T20:49:19.317437+0000","last_fullsized":"2024-09-15T20:49:19.317437+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:43:04.920505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327733+0000","last_change":"2024-09-15T20:32:49.935233+0000","last_active":"2024-09-15T20:49:20.327733+0000","last_peered":"2024-09-15T20:49:20.327733+0000","last_clean":"2024-09-15T20:49:20.327733+0000","last_became_active":"2024-09-15T20:32:49.930427+0000","last_became_peered":"2024-09-15T20:32:49.930427+0000","last_unstale":"2024-09-15T20:49:20.327733+0000","last_undegraded":"2024-09-15T20:49:20.327733+0000","last_fullsized":"2024-09-15T20:49:20.327733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:00:31.898076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317298+0000","last_change":"2024-09-15T20:32:49.880288+0000","last_active":"2024-09-15T20:49:19.317298+0000","last_peered":"2024-09-15T20:49:19.317298+0000","last_clean":"2024-09-15T20:49:19.317298+0000","last_became_active":"2024-09-15T20:32:49.880025+0000","last_became_peered":"2024-09-15T20:32:49.880025+0000","last_unstale":"2024-09-15T20:49:19.317298+0000","last_undegraded":"2024-09-15T20:49:19.317298+0000","last_fullsized":"2024-09-15T20:49:19.317298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:36.755630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327939+0000","last_change":"2024-09-15T20:32:49.942013+0000","last_active":"2024-09-15T20:49:20.327939+0000","last_peered":"2024-09-15T20:49:20.327939+0000","last_clean":"2024-09-15T20:49:20.327939+0000","last_became_active":"2024-09-15T20:32:49.939238+0000","last_became_peered":"2024-09-15T20:32:49.939238+0000","last_unstale":"2024-09-15T20:49:20.327939+0000","last_undegraded":"2024-09-15T20:49:20.327939+0000","last_fullsized":"2024-09-15T20:49:20.327939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:23:42.080704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317993+0000","last_change":"2024-09-15T20:32:49.906455+0000","last_active":"2024-09-15T20:49:19.317993+0000","last_peered":"2024-09-15T20:49:19.317993+0000","last_clean":"2024-09-15T20:49:19.317993+0000","last_became_active":"2024-09-15T20:32:49.903971+0000","last_became_peered":"2024-09-15T20:32:49.903971+0000","last_unstale":"2024-09-15T20:49:19.317993+0000","last_undegraded":"2024-09-15T20:49:19.317993+0000","last_fullsized":"2024-09-15T20:49:19.317993+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:03:18.879453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173724+0000","last_change":"2024-09-15T20:32:49.910696+0000","last_active":"2024-09-15T20:49:18.173724+0000","last_peered":"2024-09-15T20:49:18.173724+0000","last_clean":"2024-09-15T20:49:18.173724+0000","last_became_active":"2024-09-15T20:32:49.909018+0000","last_became_peered":"2024-09-15T20:32:49.909018+0000","last_unstale":"2024-09-15T20:49:18.173724+0000","last_undegraded":"2024-09-15T20:49:18.173724+0000","last_fullsized":"2024-09-15T20:49:18.173724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:03.321005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1326,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327891+0000","last_change":"2024-09-15T20:32:49.943386+0000","last_active":"2024-09-15T20:49:20.327891+0000","last_peered":"2024-09-15T20:49:20.327891+0000","last_clean":"2024-09-15T20:49:20.327891+0000","last_became_active":"2024-09-15T20:32:49.942857+0000","last_became_peered":"2024-09-15T20:32:49.942857+0000","last_unstale":"2024-09-15T20:49:20.327891+0000","last_undegraded":"2024-09-15T20:49:20.327891+0000","last_fullsized":"2024-09-15T20:49:20.327891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:01.229283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607782,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72548,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299292,"statfs":{"total":96636764160,"available":96562475008,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607781,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5256,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5382144,"data_stored":2198026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607782,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50336,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94321504,"statfs":{"total":96636764160,"available":96585220096,"internally_reserved":0,"allocated":5386240,"data_stored":2198031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607781,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72604,"kb_used_data":5320,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299236,"statfs":{"total":96636764160,"available":96562417664,"internally_reserved":0,"allocated":5447680,"data_stored":2726959,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68440,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303400,"statfs":{"total":96636764160,"available":96566681600,"internally_reserved":0,"allocated":5443584,"data_stored":2726954,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640486,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72548,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299292,"statfs":{"total":96636764160,"available":96562475008,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:50:38.497 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-09-15T20:50:38.617 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.613+0000 7f8e407e3640 10 monclient: get_monmap_and_config 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.613+0000 7f8e407e3640 10 monclient: build_initial_monmap 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.613+0000 7f8e407e3640 10 monclient: monmap: 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:38.619827+0000 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:38.619827+0000 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:38.618 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:38.619 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:38.619 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.613+0000 7f8e407e3640 1 Processor -- start 2024-09-15T20:50:38.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.613+0000 7f8e407e3640 1 -- start start 2024-09-15T20:50:38.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.613+0000 7f8e407e3640 10 monclient: init 2024-09-15T20:50:38.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:38.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:38.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): picked mon.noname-c con 0x7f8e38137d60 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:38.626 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): picked mon.noname-b con 0x7f8e38074500 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): picked mon.noname-a con 0x7f8e3813e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e38138850 con 0x7f8e3813e370 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e380a5da0 con 0x7f8e38137d60 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e38073560 con 0x7f8e38074500 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): authenticate will time out at 2299.504551s 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3ed80640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8e3813e370 0x7f8e38140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46222/0 (socket says 172.21.15.78:46222) 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3dd7e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f8e38074500 0x7f8e3813cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35454/0 (socket says 172.21.15.78:35454) 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3ed80640 1 -- 172.21.15.78:0/4101826188 learned_addr learned my addr 172.21.15.78:0/4101826188 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 116120720 0 0) 0x7f8e38138850 con 0x7f8e3813e370 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.627 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient(hunting): my global_id is 144330 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e20003530 con 0x7f8e3813e370 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3435003654 0 0) 0x7f8e38073560 con 0x7f8e38074500 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient(hunting): my global_id is 154250 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e38138850 con 0x7f8e38074500 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 898424046 0 0) 0x7f8e20003530 con 0x7f8e3813e370 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 >> v1:172.21.15.78:6790/0 conn(0x7f8e38074500 legacy=0x7f8e3813cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 >> v1:172.21.15.146:6789/0 conn(0x7f8e38137d60 legacy=0x7f8e38138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: _finish_hunting 0 2024-09-15T20:50:38.628 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 monclient: found mon.noname-a 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e38073840 con 0x7f8e3813e370 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: _finish_auth 0 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: _check_auth_tickets 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8e28001a20 con 0x7f8e3813e370 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: dump: 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.629 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.630 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.630 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.630 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.630 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.630 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.630 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 5 monclient: authenticate success, global_id 144330 2024-09-15T20:50:38.630 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e28001bc0 con 0x7f8e3813e370 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 1 -- 172.21.15.78:0/4101826188 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8e28001fd0 con 0x7f8e3813e370 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 4 set_mon_vals no callback set 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3d57d640 10 monclient: dump: 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.631 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: shutdown 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- 172.21.15.78:0/4101826188 >> v1:172.21.15.78:6789/0 conn(0x7f8e3813e370 legacy=0x7f8e38140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- 172.21.15.78:0/4101826188 shutdown_connections 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- 172.21.15.78:0/4101826188 >> 172.21.15.78:0/4101826188 conn(0x7f8e380aa650 msgr2=0x7f8e380aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- 172.21.15.78:0/4101826188 shutdown_connections 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- 172.21.15.78:0/4101826188 wait complete. 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: build_initial_monmap 2024-09-15T20:50:38.632 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: monmap: 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 librados: starting msgr at 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 librados: starting objecter 2024-09-15T20:50:38.633 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 Processor -- start 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- start start 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 librados: setting wanted keys 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 librados: calling monclient init 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: init 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): picked mon.noname-b con 0x7f8e38074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): picked mon.noname-c con 0x7f8e38137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): picked mon.noname-a con 0x7f8e3813e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e380a0b00 con 0x7f8e3813e370 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e380a0650 con 0x7f8e38074500 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e38132520 con 0x7f8e38137d60 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:38.634 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e407e3640 10 monclient(hunting): authenticate will time out at 2299.507549s 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3ed80640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f8e3813e370 0x7f8e3809f5a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46228/0 (socket says 172.21.15.78:46228) 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e3ed80640 1 -- 172.21.15.78:0/4138946913 learned_addr learned my addr 172.21.15.78:0/4138946913 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 client.?.objecter ms_handle_connect 0x7f8e3813e370 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 client.?.objecter ms_handle_connect 0x7f8e38137d60 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1599086083 0 0) 0x7f8e380a0b00 con 0x7f8e3813e370 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 monclient(hunting): my global_id is 144333 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e24004ec0 con 0x7f8e3813e370 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2841673920 0 0) 0x7f8e38132520 con 0x7f8e38137d60 2024-09-15T20:50:38.635 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.617+0000 7f8e36ffd640 10 monclient(hunting): my global_id is 154253 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e380a0b00 con 0x7f8e38137d60 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 client.?.objecter ms_handle_connect 0x7f8e38074500 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 617465173 0 0) 0x7f8e380a0b00 con 0x7f8e38137d60 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e38132520 con 0x7f8e38137d60 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8e2c0013c0 con 0x7f8e38137d60 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): dump: 2024-09-15T20:50:38.636 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 513674362 0 0) 0x7f8e380a0650 con 0x7f8e38074500 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:38.637 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): my global_id is 164149 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e380a0b00 con 0x7f8e38074500 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 775750482 0 0) 0x7f8e24004ec0 con 0x7f8e3813e370 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f8e380a0650 con 0x7f8e3813e370 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8e28001f10 con 0x7f8e3813e370 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient(hunting): dump: 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.638 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1255312752 0 0) 0x7f8e38132520 con 0x7f8e38137d60 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 >> v1:172.21.15.146:6789/0 conn(0x7f8e38074500 legacy=0x7f8e3809e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 >> v1:172.21.15.78:6789/0 conn(0x7f8e3813e370 legacy=0x7f8e3809f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient: _finish_hunting 0 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 monclient: found mon.c 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e380a00a0 con 0x7f8e38137d60 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient: _finish_auth 0 2024-09-15T20:50:38.639 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient: _check_auth_tickets 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 5 monclient: authenticate success, global_id 154253 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 10 monclient: _renew_subs 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f8e381d5e10 con 0x7f8e38137d60 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 10 client.154253.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 10 monclient: _renew_subs 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f8e381d61b0 con 0x7f8e38137d60 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 1 librados: init done 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f8e2c0047f0 con 0x7f8e38137d60 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f8e2c004b40 con 0x7f8e38137d60 2024-09-15T20:50:38.640 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e3cd7c640 4 set_mon_vals no callback set 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 monclient: dump: 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:38.641 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e18005090 con 0x7f8e38137d60 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f8e2c010bb0 con 0x7f8e38137d60 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f8e2c04b4c0 con 0x7f8e38137d60 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 client.154253.objecter ms_dispatch 0x7f8e38000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 3 client.154253.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 3 client.154253.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 20 client.154253.objecter dump_active .. 0 homeless 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e3e57f640 10 monclient: get_auth_request con 0x7f8e2403e2b0 auth_method 0 2024-09-15T20:50:38.642 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.621+0000 7f8e36ffd640 10 client.154253.objecter ms_handle_connect 0x7f8e2403e2b0 2024-09-15T20:50:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.625+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f8e2c016aa0 con 0x7f8e38137d60 2024-09-15T20:50:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.625+0000 7f8e36ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:38.643 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.625+0000 7f8e36ffd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:38.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.793+0000 7f8e407e3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:50:38.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.793+0000 7f8e407e3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:50:38.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.793+0000 7f8e407e3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:38.797 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.793+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e18005380 con 0x7f8e38137d60 2024-09-15T20:50:38.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.793+0000 7f8e36ffd640 1 -- 172.21.15.78:0/4138946913 <== mon.2 v1:172.21.15.78:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v703) ==== 74+0+13932 (unknown 581356798 0 3698185141) 0x7f8e2c0163a0 con 0x7f8e38137d60 2024-09-15T20:50:38.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.793+0000 7f8e36ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-09-15T20:50:38.798 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.793+0000 7f8e36ffd640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:50:38.798 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:50:38.798 INFO:teuthology.orchestra.run.smithi078.stdout:{"epoch":703,"fsid":"a8a3eb37-3a92-4568-8aad-3536499064e4","created":"2024-09-15T20:31:27.702188+0000","modified":"2024-09-15T20:49:18.164287+0000","last_up_change":"2024-09-15T20:31:37.715594+0000","last_in_change":"2024-09-15T20:31:29.437051+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":98,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":340,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-15T20:31:38.725860+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":6.059999942779541,"score_stable":6.059999942779541,"optimal_score":0.33000001311302185,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-15T20:31:44.650592+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":19,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":9,"pool_name":"ceph_test_rados_api_asio","create_time":"2024-09-15T20:32:48.671370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9003b16b-6a87-4659-8436-ff541ea8dfcb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6804","nonce":3648339548}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6805","nonce":3648339548}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6807","nonce":3648339548}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6806","nonce":3648339548}]},"public_addr":"172.21.15.78:6804/3648339548","cluster_addr":"172.21.15.78:6805/3648339548","heartbeat_back_addr":"172.21.15.78:6807/3648339548","heartbeat_front_addr":"172.21.15.78:6806/3648339548","state":["exists","up"]},{"osd":1,"uuid":"be602c68-218b-4573-8d19-eef568af9d4e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6800","nonce":4006351464}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6801","nonce":4006351464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6803","nonce":4006351464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6802","nonce":4006351464}]},"public_addr":"172.21.15.78:6800/4006351464","cluster_addr":"172.21.15.78:6801/4006351464","heartbeat_back_addr":"172.21.15.78:6803/4006351464","heartbeat_front_addr":"172.21.15.78:6802/4006351464","state":["exists","up"]},{"osd":2,"uuid":"1fab92e2-4880-4f66-86f9-1c59752850a1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6808","nonce":404661851}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6809","nonce":404661851}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6811","nonce":404661851}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.78:6810","nonce":404661851}]},"public_addr":"172.21.15.78:6808/404661851","cluster_addr":"172.21.15.78:6809/404661851","heartbeat_back_addr":"172.21.15.78:6811/404661851","heartbeat_front_addr":"172.21.15.78:6810/404661851","state":["exists","up"]},{"osd":3,"uuid":"74c2a9ff-0635-4664-9dc5-d156f4b60bad","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":11,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6800","nonce":490126554}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6801","nonce":490126554}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6803","nonce":490126554}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6802","nonce":490126554}]},"public_addr":"172.21.15.146:6800/490126554","cluster_addr":"172.21.15.146:6801/490126554","heartbeat_back_addr":"172.21.15.146:6803/490126554","heartbeat_front_addr":"172.21.15.146:6802/490126554","state":["exists","up"]},{"osd":4,"uuid":"0abc805d-20bc-40e6-a25a-0d9694ace9eb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6808","nonce":2960966471}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6809","nonce":2960966471}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6811","nonce":2960966471}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6810","nonce":2960966471}]},"public_addr":"172.21.15.146:6808/2960966471","cluster_addr":"172.21.15.146:6809/2960966471","heartbeat_back_addr":"172.21.15.146:6811/2960966471","heartbeat_front_addr":"172.21.15.146:6810/2960966471","state":["exists","up"]},{"osd":5,"uuid":"08c34b4f-deb1-401a-aab7-00854ae6712f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":12,"up_thru":697,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6804","nonce":516207226}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6805","nonce":516207226}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6807","nonce":516207226}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.146:6806","nonce":516207226}]},"public_addr":"172.21.15.146:6804/516207226","cluster_addr":"172.21.15.146:6805/516207226","heartbeat_back_addr":"172.21.15.146:6807/516207226","heartbeat_front_addr":"172.21.15.146:6806/516207226","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.779852+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.576609+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.081865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.582230+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:36.088303+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540719139924082687,"old_weight":0,"last_purged_snaps_scrub":"2024-09-15T20:31:35.878442+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":14,"snaps":[{"begin":4,"length":3}]},{"pool":15,"snaps":[{"begin":1,"length":4}]},{"pool":33,"snaps":[{"begin":1,"length":1}]},{"pool":67,"snaps":[{"begin":1,"length":1}]},{"pool":91,"snaps":[{"begin":4,"length":2},{"begin":7,"length":2}]},{"pool":107,"snaps":[{"begin":1,"length":1}]},{"pool":123,"snaps":[{"begin":1,"length":2}]},{"pool":139,"snaps":[{"begin":3,"length":1}]},{"pool":144,"snaps":[{"begin":3,"length":2}]},{"pool":148,"snaps":[{"begin":2,"length":1}]},{"pool":154,"snaps":[{"begin":2,"length":2}]},{"pool":163,"snaps":[{"begin":2,"length":3}]},{"pool":168,"snaps":[{"begin":3,"length":4}]},{"pool":171,"snaps":[{"begin":2,"length":4}]},{"pool":173,"snaps":[{"begin":2,"length":2}]},{"pool":183,"snaps":[{"begin":2,"length":1}]},{"pool":188,"snaps":[{"begin":1,"length":4}]},{"pool":200,"snaps":[{"begin":7,"length":1}]},{"pool":204,"snaps":[{"begin":4,"length":4}]},{"pool":224,"snaps":[{"begin":8,"length":1}]},{"pool":237,"snaps":[{"begin":10,"length":1}]},{"pool":245,"snaps":[{"begin":11,"length":2}]},{"pool":257,"snaps":[{"begin":14,"length":1}]},{"pool":321,"snaps":[{"begin":2,"length":1}]},{"pool":322,"snaps":[{"begin":3,"length":1}]},{"pool":325,"snaps":[{"begin":4,"length":1}]},{"pool":329,"snaps":[{"begin":6,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-15T20:50:38.800 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 10 librados: watch_flush enter 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 10 librados: watch_flush exit 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 20 client.154253.objecter shutdown clearing up homeless session... 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 10 client.154253.objecter successfully canceled tick 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 >> v1:172.21.15.146:6812/435598642 conn(0x7f8e2403e2b0 legacy=0x7f8e24040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 10 monclient: shutdown 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 >> v1:172.21.15.78:6790/0 conn(0x7f8e38137d60 legacy=0x7f8e3809ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 shutdown_connections 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 >> 172.21.15.78:0/4138946913 conn(0x7f8e380aa650 msgr2=0x7f8e380a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 shutdown_connections 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 1 -- 172.21.15.78:0/4138946913 wait complete. 2024-09-15T20:50:38.801 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:38.797+0000 7f8e407e3640 1 librados: shutdown 2024-09-15T20:50:39.816 INFO:tasks.ceph:Scrubbing osd.0 2024-09-15T20:50:39.817 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.949+0000 7ff843c6c640 10 monclient: get_monmap_and_config 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.949+0000 7ff843c6c640 10 monclient: build_initial_monmap 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.949+0000 7ff843c6c640 10 monclient: monmap: 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:39.956110+0000 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:39.956110+0000 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:39.954 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:39.955 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:39.955 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 Processor -- start 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- start start 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient: init 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient(hunting): picked mon.noname-c con 0x7ff83c1359a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient(hunting): picked mon.noname-b con 0x7ff83c09f5c0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient(hunting): picked mon.noname-a con 0x7ff83c13c140 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83c0a1d30 con 0x7ff83c13c140 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83c06e9c0 con 0x7ff83c1359a0 2024-09-15T20:50:39.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83c06eca0 con 0x7ff83c09f5c0 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient(hunting): authenticate will time out at 2300.841187s 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff841207640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff83c09f5c0 0x7ff83c13aa20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35484/0 (socket says 172.21.15.78:35484) 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff841207640 1 -- 172.21.15.78:0/103172985 learned_addr learned my addr 172.21.15.78:0/103172985 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 176227269 0 0) 0x7ff83c0a1d30 con 0x7ff83c13c140 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient(hunting): _negotiate 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient(hunting): my global_id is 144336 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff828003530 con 0x7ff83c13c140 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2752830468 0 0) 0x7ff83c06eca0 con 0x7ff83c09f5c0 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient(hunting): _negotiate 2024-09-15T20:50:39.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient(hunting): my global_id is 154256 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff83c0a1d30 con 0x7ff83c09f5c0 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3745871488 0 0) 0x7ff828003530 con 0x7ff83c13c140 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 >> v1:172.21.15.78:6790/0 conn(0x7ff83c09f5c0 legacy=0x7ff83c13aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 >> v1:172.21.15.146:6789/0 conn(0x7ff83c1359a0 legacy=0x7ff83c135d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: _finish_hunting 0 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 monclient: found mon.noname-a 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83c135700 con 0x7ff83c13c140 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: _finish_auth 0 2024-09-15T20:50:39.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: _check_auth_tickets 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff834001a20 con 0x7ff83c13c140 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 5 monclient: authenticate success, global_id 144336 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: dump: 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.967 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff834001c40 con 0x7ff83c13c140 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 1 -- 172.21.15.78:0/103172985 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff834002050 con 0x7ff83c13c140 2024-09-15T20:50:39.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 4 set_mon_vals no callback set 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff840a06640 10 monclient: dump: 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:39.969 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient: shutdown 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- 172.21.15.78:0/103172985 >> v1:172.21.15.78:6789/0 conn(0x7ff83c13c140 legacy=0x7ff83c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- 172.21.15.78:0/103172985 shutdown_connections 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- 172.21.15.78:0/103172985 >> 172.21.15.78:0/103172985 conn(0x7ff83c0a6240 msgr2=0x7ff83c0a6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- 172.21.15.78:0/103172985 shutdown_connections 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 -- 172.21.15.78:0/103172985 wait complete. 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient: build_initial_monmap 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 10 monclient: monmap: 2024-09-15T20:50:39.970 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:39.971 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 librados: starting msgr at 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 librados: starting objecter 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.953+0000 7ff843c6c640 1 Processor -- start 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 -- start start 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 librados: setting wanted keys 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 librados: calling monclient init 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient: init 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient(hunting): picked mon.noname-b con 0x7ff83c09f5c0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient(hunting): picked mon.noname-a con 0x7ff83c1359a0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:39.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient(hunting): picked mon.noname-c con 0x7ff83c13c140 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83c138e50 con 0x7ff83c1359a0 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83c13ae50 con 0x7ff83c09f5c0 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff83c139960 con 0x7ff83c13c140 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient(hunting): authenticate will time out at 2300.844540s 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff841207640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff83c1359a0 0x7ff83c138740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46256/0 (socket says 172.21.15.78:46256) 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff841207640 1 -- 172.21.15.78:0/1658502552 learned_addr learned my addr 172.21.15.78:0/1658502552 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff842209640 1 --1- 172.21.15.78:0/1658502552 >> v1:172.21.15.78:6790/0 conn(0x7ff83c13c140 0x7ff83c1d2b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35486/0 (socket says 172.21.15.78:35486) 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.?.objecter ms_handle_connect 0x7ff83c1359a0 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.?.objecter ms_handle_connect 0x7ff83c13c140 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:39.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 353979468 0 0) 0x7ff83c138e50 con 0x7ff83c1359a0 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): my global_id is 144339 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff820004ec0 con 0x7ff83c1359a0 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 303041749 0 0) 0x7ff83c139960 con 0x7ff83c13c140 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): my global_id is 154259 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff83c138e50 con 0x7ff83c13c140 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.?.objecter ms_handle_connect 0x7ff83c09f5c0 2024-09-15T20:50:39.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4138910336 0 0) 0x7ff83c138e50 con 0x7ff83c13c140 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff83c139960 con 0x7ff83c13c140 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff834001f90 con 0x7ff83c13c140 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): dump: 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:39.975 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1867348065 0 0) 0x7ff820004ec0 con 0x7ff83c1359a0 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff83c138e50 con 0x7ff83c1359a0 2024-09-15T20:50:39.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff82c0013c0 con 0x7ff83c1359a0 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): dump: 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:39.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2550630642 0 0) 0x7ff83c13ae50 con 0x7ff83c09f5c0 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient(hunting): my global_id is 164155 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff820004ec0 con 0x7ff83c09f5c0 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 2216341427 0 0) 0x7ff83c139960 con 0x7ff83c13c140 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 >> v1:172.21.15.146:6789/0 conn(0x7ff83c09f5c0 legacy=0x7ff83c138030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 >> v1:172.21.15.78:6789/0 conn(0x7ff83c1359a0 legacy=0x7ff83c138740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: _finish_hunting 0 2024-09-15T20:50:39.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 monclient: found mon.c 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff83c11c910 con 0x7ff83c13c140 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: _finish_auth 0 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: _check_auth_tickets 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 5 monclient: authenticate success, global_id 154259 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient: _renew_subs 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff83c1352e0 con 0x7ff83c13c140 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 client.154259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient: _renew_subs 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:39.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7ff83c13a720 con 0x7ff83c13c140 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 librados: init done 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff834003cf0 con 0x7ff83c13c140 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff834004140 con 0x7ff83c13c140 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff833fff640 4 set_mon_vals no callback set 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: dump: 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.980 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 client.154259.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 15 client.154259.objecter _session_command_op_assign -1 1 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 20 client.154259.objecter _get_session osd=-1 returning homeless 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 client.154259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7ff824002920 con 0x7ff83c13c140 2024-09-15T20:50:39.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff8340042e0 con 0x7ff83c13c140 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7ff83404b680 con 0x7ff83c13c140 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.154259.objecter ms_dispatch 0x7ff83c000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 3 client.154259.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 3 client.154259.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.154259.objecter checking command 1 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 client.154259.objecter _get_session s=0x7ff8200460b0 osd=0 3 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 client.154259.objecter put_session s=0x7ff8200460b0 osd=0 3 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 15 client.154259.objecter _session_command_op_remove -1 1 2024-09-15T20:50:39.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 client.154259.objecter _get_session s=0x7ff8200460b0 osd=0 3 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 client.154259.objecter get_session s=0x7ff8200460b0 osd=0 3 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 15 client.154259.objecter _session_command_op_assign 0 1 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 client.154259.objecter put_session s=0x7ff8200460b0 osd=0 4 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.154259.objecter _send_command 1 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6804/3648339548 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff82004a470 con 0x7ff820046760 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 client.154259.objecter dump_active .. 0 homeless 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 3222990830 0 0) 0x7ff83404ba10 con 0x7ff83c13c140 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff841207640 10 monclient: get_auth_request con 0x7ff820046760 auth_method 0 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff841a08640 10 monclient: get_auth_request con 0x7ff82003e220 auth_method 0 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.154259.objecter ms_handle_connect 0x7ff820046760 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.154259.objecter ms_handle_connect 0x7ff82003e220 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== osd.0 v1:172.21.15.78:6804/3648339548 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7ff82004a470 con 0x7ff820046760 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.154259.objecter ms_dispatch 0x7ff83c000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:50:39.983 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 10 client.154259.objecter _finish_command 1 = system:0 2024-09-15T20:50:39.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 20 client.154259.objecter put_session s=0x7ff8200460b0 osd=0 4 2024-09-15T20:50:39.984 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.957+0000 7ff8327fc640 15 client.154259.objecter _session_command_op_remove 0 1 2024-09-15T20:50:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 10 client.154259.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T20:50:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 15 client.154259.objecter _session_command_op_assign -1 2 2024-09-15T20:50:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 20 client.154259.objecter _get_session s=0x7ff8200460b0 osd=0 3 2024-09-15T20:50:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 20 client.154259.objecter put_session s=0x7ff8200460b0 osd=0 3 2024-09-15T20:50:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 20 client.154259.objecter _get_session s=0x7ff8200460b0 osd=0 3 2024-09-15T20:50:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 15 client.154259.objecter _session_command_op_remove -1 2 2024-09-15T20:50:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 20 client.154259.objecter get_session s=0x7ff8200460b0 osd=0 3 2024-09-15T20:50:39.985 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 15 client.154259.objecter _session_command_op_assign 0 2 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 20 client.154259.objecter put_session s=0x7ff8200460b0 osd=0 4 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 10 client.154259.objecter _send_command 2 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 --> v1:172.21.15.78:6804/3648339548 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff8240012c0 con 0x7ff820046760 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff8327fc640 1 -- 172.21.15.78:0/1658502552 <== osd.0 v1:172.21.15.78:6804/3648339548 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7ff8240012c0 con 0x7ff820046760 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff8327fc640 10 client.154259.objecter ms_dispatch 0x7ff83c000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff8327fc640 10 client.154259.objecter _finish_command 2 = system:0 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff8327fc640 20 client.154259.objecter put_session s=0x7ff8200460b0 osd=0 4 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff8327fc640 15 client.154259.objecter _session_command_op_remove 0 2 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:39.986 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:50:39.987 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 10 librados: watch_flush enter 2024-09-15T20:50:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 10 librados: watch_flush exit 2024-09-15T20:50:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 10 client.154259.objecter close_session for osd.0 2024-09-15T20:50:39.988 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 >> v1:172.21.15.78:6804/3648339548 conn(0x7ff820046760 legacy=0x7ff820048c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 20 client.154259.objecter put_session s=0x7ff8200460b0 osd=0 1 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 20 client.154259.objecter shutdown clearing up homeless session... 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 10 client.154259.objecter successfully canceled tick 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 >> v1:172.21.15.146:6812/435598642 conn(0x7ff82003e220 legacy=0x7ff8200406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 10 monclient: shutdown 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 >> v1:172.21.15.78:6790/0 conn(0x7ff83c13c140 legacy=0x7ff83c1d2b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff842209640 1 -- 172.21.15.78:0/1658502552 reap_dead start 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 shutdown_connections 2024-09-15T20:50:39.989 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 >> 172.21.15.78:0/1658502552 conn(0x7ff83c0a6240 msgr2=0x7ff83c13f330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:39.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 shutdown_connections 2024-09-15T20:50:39.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 -- 172.21.15.78:0/1658502552 wait complete. 2024-09-15T20:50:39.990 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:39.973+0000 7ff843c6c640 1 librados: shutdown 2024-09-15T20:50:39.991 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-09-15T20:50:40.106 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.101+0000 7feaa7758640 10 monclient: get_monmap_and_config 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.101+0000 7feaa7758640 10 monclient: build_initial_monmap 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient: monmap: 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:40.108901+0000 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:40.108901+0000 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:40.107 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 Processor -- start 2024-09-15T20:50:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- start start 2024-09-15T20:50:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient: init 2024-09-15T20:50:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:40.114 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient(hunting): picked mon.noname-b con 0x7feaa0137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient(hunting): picked mon.noname-a con 0x7feaa0074500 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient(hunting): picked mon.noname-c con 0x7feaa013e370 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa0138850 con 0x7feaa0074500 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa00a5da0 con 0x7feaa013e370 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa0073560 con 0x7feaa0137d60 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient(hunting): authenticate will time out at 2300.993446s 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa4cf3640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7feaa0074500 0x7feaa013cc50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46260/0 (socket says 172.21.15.78:46260) 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa4cf3640 1 -- 172.21.15.78:0/1831091361 learned_addr learned my addr 172.21.15.78:0/1831091361 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa54f4640 1 --1- 172.21.15.78:0/1831091361 >> v1:172.21.15.78:6790/0 conn(0x7feaa0137d60 0x7feaa0138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:35496/0 (socket says 172.21.15.78:35496) 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 749296509 0 0) 0x7feaa0138850 con 0x7feaa0074500 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.115 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient(hunting): my global_id is 144342 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea8c003530 con 0x7feaa0074500 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2894860642 0 0) 0x7feaa0073560 con 0x7feaa0137d60 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient(hunting): my global_id is 154262 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaa0138850 con 0x7feaa0137d60 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2071881877 0 0) 0x7fea8c003530 con 0x7feaa0074500 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 >> v1:172.21.15.78:6790/0 conn(0x7feaa0137d60 legacy=0x7feaa0138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 >> v1:172.21.15.146:6789/0 conn(0x7feaa013e370 legacy=0x7feaa0140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: _finish_hunting 0 2024-09-15T20:50:40.116 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 monclient: found mon.noname-a 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa0073840 con 0x7feaa0074500 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: _finish_auth 0 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: _check_auth_tickets 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fea900016a0 con 0x7feaa0074500 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 5 monclient: authenticate success, global_id 144342 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: dump: 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.117 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea900018c0 con 0x7feaa0074500 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 4 set_mon_vals no callback set 2024-09-15T20:50:40.118 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 1 -- 172.21.15.78:0/1831091361 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fea90001c90 con 0x7feaa0074500 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7fea97fff640 10 monclient: dump: 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.119 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient: shutdown 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- 172.21.15.78:0/1831091361 >> v1:172.21.15.78:6789/0 conn(0x7feaa0074500 legacy=0x7feaa013cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- 172.21.15.78:0/1831091361 shutdown_connections 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- 172.21.15.78:0/1831091361 >> 172.21.15.78:0/1831091361 conn(0x7feaa00aa650 msgr2=0x7feaa00aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- 172.21.15.78:0/1831091361 shutdown_connections 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 -- 172.21.15.78:0/1831091361 wait complete. 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient: build_initial_monmap 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 10 monclient: monmap: 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:40.120 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 librados: starting msgr at 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 librados: starting objecter 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.105+0000 7feaa7758640 1 Processor -- start 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 -- start start 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 librados: setting wanted keys 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 librados: calling monclient init 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: init 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:40.121 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient(hunting): picked mon.noname-b con 0x7feaa0074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient(hunting): picked mon.noname-c con 0x7feaa0137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient(hunting): picked mon.noname-a con 0x7feaa013e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa0132b20 con 0x7feaa013e370 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa01405c0 con 0x7feaa0074500 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feaa00a3fd0 con 0x7feaa0137d60 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient(hunting): authenticate will time out at 2300.996532s 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa5cf5640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7feaa013e370 0x7feaa01d4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:46266/0 (socket says 172.21.15.78:46266) 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa5cf5640 1 -- 172.21.15.78:0/469165887 learned_addr learned my addr 172.21.15.78:0/469165887 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 client.?.objecter ms_handle_connect 0x7feaa013e370 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 client.?.objecter ms_handle_connect 0x7feaa0137d60 2024-09-15T20:50:40.122 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1031096924 0 0) 0x7feaa0132b20 con 0x7feaa013e370 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): my global_id is 144345 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea84004ec0 con 0x7feaa013e370 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 157021540 0 0) 0x7feaa00a3fd0 con 0x7feaa0137d60 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): my global_id is 154265 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7feaa0132b20 con 0x7feaa0137d60 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 client.?.objecter ms_handle_connect 0x7feaa0074500 2024-09-15T20:50:40.123 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 427412194 0 0) 0x7feaa0132b20 con 0x7feaa0137d60 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feaa00a3fd0 con 0x7feaa0137d60 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fea90001a70 con 0x7feaa0137d60 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): dump: 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.124 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1819249229 0 0) 0x7fea84004ec0 con 0x7feaa013e370 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7feaa0132b20 con 0x7feaa013e370 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fea98001cc0 con 0x7feaa013e370 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): dump: 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.125 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1018773713 0 0) 0x7feaa01405c0 con 0x7feaa0074500 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.126 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient(hunting): my global_id is 164161 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fea84004ec0 con 0x7feaa0074500 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1395436218 0 0) 0x7feaa00a3fd0 con 0x7feaa0137d60 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 >> v1:172.21.15.146:6789/0 conn(0x7feaa0074500 legacy=0x7feaa013f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 >> v1:172.21.15.78:6789/0 conn(0x7feaa013e370 legacy=0x7feaa01d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient: _finish_hunting 0 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 monclient: found mon.c 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feaa01418a0 con 0x7feaa0137d60 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient: _finish_auth 0 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient: _check_auth_tickets 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 5 monclient: authenticate success, global_id 154265 2024-09-15T20:50:40.127 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: _renew_subs 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7feaa0132840 con 0x7feaa0137d60 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 client.154265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: _renew_subs 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7feaa00a05c0 con 0x7feaa0137d60 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 librados: init done 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fea90001820 con 0x7feaa0137d60 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fea90004c00 con 0x7feaa0137d60 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 monclient: dump: 2024-09-15T20:50:40.128 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea977fe640 4 set_mon_vals no callback set 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.129 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea80005090 con 0x7feaa0137d60 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fea90010c70 con 0x7feaa0137d60 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fea9004b300 con 0x7feaa0137d60 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 client.154265.objecter ms_dispatch 0x7feaa0000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 3 client.154265.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 3 client.154265.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 20 client.154265.objecter dump_active .. 0 homeless 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7feaa54f4640 10 monclient: get_auth_request con 0x7fea8403e2c0 auth_method 0 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.109+0000 7fea95ffb640 10 client.154265.objecter ms_handle_connect 0x7fea8403e2c0 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.113+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fea90016880 con 0x7feaa0137d60 2024-09-15T20:50:40.130 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.113+0000 7fea95ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:40.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.113+0000 7fea95ffb640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.273+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7fea80000ba0 con 0x7fea8403e2c0 2024-09-15T20:50:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.273+0000 7fea95ffb640 1 -- 172.21.15.78:0/469165887 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (unknown 2057319476 0 0) 0x7fea80000ba0 con 0x7fea8403e2c0 2024-09-15T20:50:40.277 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 0 to deep-scrub 2024-09-15T20:50:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 10 librados: watch_flush enter 2024-09-15T20:50:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 10 librados: watch_flush exit 2024-09-15T20:50:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 20 client.154265.objecter shutdown clearing up homeless session... 2024-09-15T20:50:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 10 client.154265.objecter successfully canceled tick 2024-09-15T20:50:40.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 >> v1:172.21.15.146:6812/435598642 conn(0x7fea8403e2c0 legacy=0x7fea84040780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 10 monclient: shutdown 2024-09-15T20:50:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 >> v1:172.21.15.78:6790/0 conn(0x7feaa0137d60 legacy=0x7feaa013feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 shutdown_connections 2024-09-15T20:50:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 >> 172.21.15.78:0/469165887 conn(0x7feaa00aa650 msgr2=0x7feaa00a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 shutdown_connections 2024-09-15T20:50:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 1 -- 172.21.15.78:0/469165887 wait complete. 2024-09-15T20:50:40.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.277+0000 7feaa7758640 1 librados: shutdown 2024-09-15T20:50:40.295 INFO:tasks.ceph:Scrubbing osd.1 2024-09-15T20:50:40.295 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:50:40.423 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient: get_monmap_and_config 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient: build_initial_monmap 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient: monmap: 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:40.417329+0000 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:40.417329+0000 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:40.424 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 Processor -- start 2024-09-15T20:50:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- start start 2024-09-15T20:50:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient: init 2024-09-15T20:50:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:50:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient(hunting): picked mon.noname-a con 0x7fdba809d3b0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:40.425 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient(hunting): picked mon.noname-b con 0x7fdba809f330 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient(hunting): picked mon.noname-c con 0x7fdba809dea0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba809f120 con 0x7fdba809d3b0 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba809fbb0 con 0x7fdba809dea0 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba80a0cd0 con 0x7fdba809f330 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient(hunting): authenticate will time out at 2301.302284s 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbad75f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdba809d3b0 0x7fdba809d790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57870/0 (socket says 172.21.15.78:57870) 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbad75f640 1 -- 172.21.15.78:0/2163361631 learned_addr learned my addr 172.21.15.78:0/2163361631 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3326553994 0 0) 0x7fdba80a0cd0 con 0x7fdba809f330 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient(hunting): my global_id is 154268 2024-09-15T20:50:40.426 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb94003530 con 0x7fdba809f330 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1742923541 0 0) 0x7fdba809f120 con 0x7fdba809d3b0 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient(hunting): my global_id is 144348 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba80a0cd0 con 0x7fdba809d3b0 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 542057494 0 0) 0x7fdb94003530 con 0x7fdba809f330 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 >> v1:172.21.15.146:6789/0 conn(0x7fdba809dea0 legacy=0x7fdba813d200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 >> v1:172.21.15.78:6789/0 conn(0x7fdba809d3b0 legacy=0x7fdba809d790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 monclient: found mon.noname-b 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.427 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba806e5d0 con 0x7fdba809f330 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: _finish_auth 0 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 5 monclient: authenticate success, global_id 154268 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdb980016a0 con 0x7fdba809f330 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: dump: 2024-09-15T20:50:40.428 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.429 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdb980018c0 con 0x7fdba809f330 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 1 -- 172.21.15.78:0/2163361631 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdb98001d50 con 0x7fdba809f330 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 4 set_mon_vals no callback set 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdb9ffff640 10 monclient: dump: 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.430 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 10 monclient: shutdown 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2163361631 >> v1:172.21.15.78:6790/0 conn(0x7fdba809f330 legacy=0x7fdba8139ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2163361631 shutdown_connections 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2163361631 >> 172.21.15.78:0/2163361631 conn(0x7fdba80a6240 msgr2=0x7fdba80a6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2163361631 shutdown_connections 2024-09-15T20:50:40.431 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.413+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2163361631 wait complete. 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: build_initial_monmap 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: monmap: 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:40.432 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 librados: starting msgr at 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 librados: starting objecter 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 Processor -- start 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 -- start start 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 librados: setting wanted keys 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 librados: calling monclient init 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: init 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient(hunting): picked mon.noname-a con 0x7fdba809d3b0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:40.433 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient(hunting): picked mon.noname-b con 0x7fdba809dea0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient(hunting): picked mon.noname-c con 0x7fdba809f330 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba8123850 con 0x7fdba809d3b0 2024-09-15T20:50:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba8120890 con 0x7fdba809dea0 2024-09-15T20:50:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdba809d0a0 con 0x7fdba809f330 2024-09-15T20:50:40.434 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient(hunting): authenticate will time out at 2301.305408s 2024-09-15T20:50:40.435 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbad75f640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fdba809d3b0 0x7fdba811fa70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57882/0 (socket says 172.21.15.78:57882) 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbad75f640 1 -- 172.21.15.78:0/2038631174 learned_addr learned my addr 172.21.15.78:0/2038631174 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.?.objecter ms_handle_connect 0x7fdba809d3b0 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.?.objecter ms_handle_connect 0x7fdba809f330 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 144627258 0 0) 0x7fdba8123850 con 0x7fdba809d3b0 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): my global_id is 144351 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdb8c004ef0 con 0x7fdba809d3b0 2024-09-15T20:50:40.436 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.?.objecter ms_handle_connect 0x7fdba809dea0 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 763078210 0 0) 0x7fdba809d0a0 con 0x7fdba809f330 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): my global_id is 154271 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdba8123850 con 0x7fdba809f330 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 125740622 0 0) 0x7fdb8c004ef0 con 0x7fdba809d3b0 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fdba809d0a0 con 0x7fdba809d3b0 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdba40012e0 con 0x7fdba809d3b0 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:40.437 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): dump: 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3845571343 0 0) 0x7fdba809d0a0 con 0x7fdba809d3b0 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 >> v1:172.21.15.78:6790/0 conn(0x7fdba809f330 legacy=0x7fdba81d08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.438 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 >> v1:172.21.15.146:6789/0 conn(0x7fdba809dea0 legacy=0x7fdba8120180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: _finish_hunting 0 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 monclient: found mon.a 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdba81229e0 con 0x7fdba809d3b0 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: _finish_auth 0 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: _check_auth_tickets 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 5 monclient: authenticate success, global_id 144351 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: _renew_subs 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fdba81214f0 con 0x7fdba809d3b0 2024-09-15T20:50:40.439 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 client.144351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: _renew_subs 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fdba8134370 con 0x7fdba809d3b0 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 librados: init done 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fdba40015e0 con 0x7fdba809d3b0 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fdba4004c40 con 0x7fdba809d3b0 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:40.440 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: dump: 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9f7fe640 4 set_mon_vals no callback set 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 client.144351.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:40.441 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 15 client.144351.objecter _session_command_op_assign -1 1 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 20 client.144351.objecter _get_session osd=-1 returning homeless 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 client.144351.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6789/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fdb88002920 con 0x7fdba809d3b0 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fdba4010cb0 con 0x7fdba809d3b0 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fdba404b440 con 0x7fdba809d3b0 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.144351.objecter ms_dispatch 0x7fdba8000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 3 client.144351.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 3 client.144351.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.144351.objecter checking command 1 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 20 client.144351.objecter _get_session s=0x7fdb8c046010 osd=1 3 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 20 client.144351.objecter put_session s=0x7fdb8c046010 osd=1 3 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 15 client.144351.objecter _session_command_op_remove -1 1 2024-09-15T20:50:40.442 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 20 client.144351.objecter _get_session s=0x7fdb8c046010 osd=1 3 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 20 client.144351.objecter get_session s=0x7fdb8c046010 osd=1 3 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 15 client.144351.objecter _session_command_op_assign 1 1 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 20 client.144351.objecter put_session s=0x7fdb8c046010 osd=1 4 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.144351.objecter _send_command 1 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6800/4006351464 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdb8c04a3d0 con 0x7fdb8c0466c0 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 20 client.144351.objecter dump_active .. 0 homeless 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 3222990830 0 0) 0x7fdba404b910 con 0x7fdba809d3b0 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbadf60640 10 monclient: get_auth_request con 0x7fdb8c0466c0 auth_method 0 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdbacf5e640 10 monclient: get_auth_request con 0x7fdb8c03e130 auth_method 0 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.417+0000 7fdb9dffb640 10 client.144351.objecter ms_handle_connect 0x7fdb8c0466c0 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.421+0000 7fdb9dffb640 10 client.144351.objecter ms_handle_connect 0x7fdb8c03e130 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.421+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== osd.1 v1:172.21.15.78:6800/4006351464 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fdb8c04a3d0 con 0x7fdb8c0466c0 2024-09-15T20:50:40.443 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.421+0000 7fdb9dffb640 10 client.144351.objecter ms_dispatch 0x7fdba8000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:50:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.421+0000 7fdb9dffb640 10 client.144351.objecter _finish_command 1 = system:0 2024-09-15T20:50:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.421+0000 7fdb9dffb640 20 client.144351.objecter put_session s=0x7fdb8c046010 osd=1 4 2024-09-15T20:50:40.444 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.421+0000 7fdb9dffb640 15 client.144351.objecter _session_command_op_remove 1 1 2024-09-15T20:50:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 10 client.144351.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T20:50:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 15 client.144351.objecter _session_command_op_assign -1 2 2024-09-15T20:50:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 20 client.144351.objecter _get_session s=0x7fdb8c046010 osd=1 3 2024-09-15T20:50:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 20 client.144351.objecter put_session s=0x7fdb8c046010 osd=1 3 2024-09-15T20:50:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 20 client.144351.objecter _get_session s=0x7fdb8c046010 osd=1 3 2024-09-15T20:50:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 15 client.144351.objecter _session_command_op_remove -1 2 2024-09-15T20:50:40.445 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 20 client.144351.objecter get_session s=0x7fdb8c046010 osd=1 3 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 15 client.144351.objecter _session_command_op_assign 1 2 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 20 client.144351.objecter put_session s=0x7fdb8c046010 osd=1 4 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 10 client.144351.objecter _send_command 2 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 --> v1:172.21.15.78:6800/4006351464 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fdb880012c0 con 0x7fdb8c0466c0 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdb9dffb640 1 -- 172.21.15.78:0/2038631174 <== osd.1 v1:172.21.15.78:6800/4006351464 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fdb880012c0 con 0x7fdb8c0466c0 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdb9dffb640 10 client.144351.objecter ms_dispatch 0x7fdba8000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdb9dffb640 10 client.144351.objecter _finish_command 2 = system:0 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdb9dffb640 20 client.144351.objecter put_session s=0x7fdb8c046010 osd=1 4 2024-09-15T20:50:40.446 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdb9dffb640 15 client.144351.objecter _session_command_op_remove 1 2 2024-09-15T20:50:40.447 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:40.447 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:50:40.447 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 10 librados: watch_flush enter 2024-09-15T20:50:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 10 librados: watch_flush exit 2024-09-15T20:50:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 10 client.144351.objecter close_session for osd.1 2024-09-15T20:50:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 >> v1:172.21.15.78:6800/4006351464 conn(0x7fdb8c0466c0 legacy=0x7fdb8c048b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 20 client.144351.objecter put_session s=0x7fdb8c046010 osd=1 1 2024-09-15T20:50:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 20 client.144351.objecter shutdown clearing up homeless session... 2024-09-15T20:50:40.448 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 10 client.144351.objecter successfully canceled tick 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 >> v1:172.21.15.146:6812/435598642 conn(0x7fdb8c03e130 legacy=0x7fdb8c0405f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 10 monclient: shutdown 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 >> v1:172.21.15.78:6789/0 conn(0x7fdba809d3b0 legacy=0x7fdba811fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbadf60640 1 -- 172.21.15.78:0/2038631174 reap_dead start 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 shutdown_connections 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 >> 172.21.15.78:0/2038631174 conn(0x7fdba80a6240 msgr2=0x7fdba8134c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 shutdown_connections 2024-09-15T20:50:40.449 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 -- 172.21.15.78:0/2038631174 wait complete. 2024-09-15T20:50:40.450 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.433+0000 7fdbaf9c3640 1 librados: shutdown 2024-09-15T20:50:40.451 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-09-15T20:50:40.559 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient: get_monmap_and_config 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient: build_initial_monmap 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient: monmap: 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:40.561929+0000 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:40.561929+0000 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:40.560 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 Processor -- start 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- start start 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient: init 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient(hunting): picked mon.noname-a con 0x7f3ef4134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient(hunting): picked mon.noname-c con 0x7f3ef409f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient(hunting): picked mon.noname-b con 0x7f3ef406e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:40.567 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef40a0470 con 0x7f3ef4134bb0 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef409feb0 con 0x7f3ef409f5a0 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef409e4a0 con 0x7f3ef406e960 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient(hunting): authenticate will time out at 2301.446325s 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3ef4134bb0 0x7f3ef4134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57888/0 (socket says 172.21.15.78:57888) 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef37fe640 1 -- 172.21.15.78:0/3971986118 learned_addr learned my addr 172.21.15.78:0/3971986118 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 300368954 0 0) 0x7f3ee4001120 con 0x7f3ef4134bb0 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.568 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient(hunting): my global_id is 144354 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3edc003530 con 0x7f3ef4134bb0 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 532170193 0 0) 0x7f3ef409e4a0 con 0x7f3ef406e960 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient(hunting): my global_id is 154274 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ee4001120 con 0x7f3ef406e960 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3839307937 0 0) 0x7f3ee4001120 con 0x7f3ef406e960 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 >> v1:172.21.15.146:6789/0 conn(0x7f3ef409f5a0 legacy=0x7f3ef41399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 >> v1:172.21.15.78:6789/0 conn(0x7f3ef4134bb0 legacy=0x7f3ef4134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.569 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: _finish_hunting 0 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 monclient: found mon.noname-b 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef40a0ff0 con 0x7f3ef406e960 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: _finish_auth 0 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: _check_auth_tickets 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3eec001980 con 0x7f3ef406e960 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 5 monclient: authenticate success, global_id 154274 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: dump: 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.570 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3eec001ba0 con 0x7f3ef406e960 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 1 -- 172.21.15.78:0/3971986118 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3eec001ff0 con 0x7f3ef406e960 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef27fc640 10 monclient: dump: 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:40.572 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 4 set_mon_vals no callback set 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 10 monclient: shutdown 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/3971986118 >> v1:172.21.15.78:6790/0 conn(0x7f3ef406e960 legacy=0x7f3ef413d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/3971986118 shutdown_connections 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/3971986118 >> 172.21.15.78:0/3971986118 conn(0x7f3ef40a63c0 msgr2=0x7f3ef40a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/3971986118 shutdown_connections 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.557+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/3971986118 wait complete. 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: build_initial_monmap 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: monmap: 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:40.573 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 librados: starting msgr at 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 librados: starting objecter 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 Processor -- start 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 -- start start 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 librados: setting wanted keys 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 librados: calling monclient init 2024-09-15T20:50:40.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: init 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient(hunting): picked mon.noname-a con 0x7f3ef406e960 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient(hunting): picked mon.noname-b con 0x7f3ef409f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient(hunting): picked mon.noname-c con 0x7f3ef4134bb0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef40a4380 con 0x7f3ef406e960 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef409d180 con 0x7f3ef409f5a0 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ef411d400 con 0x7f3ef4134bb0 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient(hunting): authenticate will time out at 2301.449405s 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef3fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f3ef4134bb0 0x7f3ef41d4f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38480/0 (socket says 172.21.15.78:38480) 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef37fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f3ef406e960 0x7f3ef411c5e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57890/0 (socket says 172.21.15.78:57890) 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef3fff640 1 -- 172.21.15.78:0/2386948336 learned_addr learned my addr 172.21.15.78:0/2386948336 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:40.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 client.?.objecter ms_handle_connect 0x7f3ef4134bb0 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 client.?.objecter ms_handle_connect 0x7f3ef406e960 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3505803317 0 0) 0x7f3ef411d400 con 0x7f3ef4134bb0 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): my global_id is 154277 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ee0004ef0 con 0x7f3ef4134bb0 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877162985 0 0) 0x7f3ef40a4380 con 0x7f3ef406e960 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.576 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): my global_id is 144357 2024-09-15T20:50:40.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ef411d400 con 0x7f3ef406e960 2024-09-15T20:50:40.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 client.?.objecter ms_handle_connect 0x7f3ef409f5a0 2024-09-15T20:50:40.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1119159684 0 0) 0x7f3ef411d400 con 0x7f3ef406e960 2024-09-15T20:50:40.577 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ef40a4380 con 0x7f3ef406e960 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 979616716 0 0) 0x7f3ee0004ef0 con 0x7f3ef4134bb0 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3ef411d400 con 0x7f3ef4134bb0 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3ee4001b30 con 0x7f3ef406e960 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): dump: 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.578 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3eec001fb0 con 0x7f3ef4134bb0 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): dump: 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.579 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1090542785 0 0) 0x7f3ef409d180 con 0x7f3ef409f5a0 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient(hunting): my global_id is 164173 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ee0004ef0 con 0x7f3ef409f5a0 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 173286680 0 0) 0x7f3ef40a4380 con 0x7f3ef406e960 2024-09-15T20:50:40.580 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 >> v1:172.21.15.78:6790/0 conn(0x7f3ef4134bb0 legacy=0x7f3ef41d4f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 >> v1:172.21.15.146:6789/0 conn(0x7f3ef409f5a0 legacy=0x7f3ef411ccf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient: _finish_hunting 0 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 monclient: found mon.a 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ef411eab0 con 0x7f3ef406e960 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient: _finish_auth 0 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient: _check_auth_tickets 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 5 monclient: authenticate success, global_id 144357 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: _renew_subs 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3ef411ed90 con 0x7f3ef406e960 2024-09-15T20:50:40.581 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 client.144357.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: _renew_subs 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f3ef411d870 con 0x7f3ef406e960 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 librados: init done 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3ee40047f0 con 0x7f3ef406e960 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3ee4004b80 con 0x7f3ef406e960 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef1ffb640 4 set_mon_vals no callback set 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 monclient: dump: 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.582 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ed4005090 con 0x7f3ef406e960 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f3ee4010bf0 con 0x7f3ef406e960 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f3ee404b3a0 con 0x7f3ef406e960 2024-09-15T20:50:40.583 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 10 client.144357.objecter ms_dispatch 0x7f3ef4000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:40.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 3 client.144357.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:40.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 3 client.144357.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:40.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3edbfff640 20 client.144357.objecter dump_active .. 0 homeless 2024-09-15T20:50:40.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.561+0000 7f3ef2ffd640 10 monclient: get_auth_request con 0x7f3ee003e250 auth_method 0 2024-09-15T20:50:40.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.565+0000 7f3edbfff640 10 client.144357.objecter ms_handle_connect 0x7f3ee003e250 2024-09-15T20:50:40.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.565+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f3ef40a0470 con 0x7f3ef406e960 2024-09-15T20:50:40.584 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.565+0000 7f3edbfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:40.585 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.565+0000 7f3edbfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:40.739 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7f3ed4000ba0 con 0x7f3ee003e250 2024-09-15T20:50:40.740 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3edbfff640 1 -- 172.21.15.78:0/2386948336 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (unknown 489605429 0 0) 0x7f3ed4000ba0 con 0x7f3ee003e250 2024-09-15T20:50:40.740 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 1 to deep-scrub 2024-09-15T20:50:40.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 10 librados: watch_flush enter 2024-09-15T20:50:40.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 10 librados: watch_flush exit 2024-09-15T20:50:40.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 20 client.144357.objecter shutdown clearing up homeless session... 2024-09-15T20:50:40.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 10 client.144357.objecter successfully canceled tick 2024-09-15T20:50:40.742 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 >> v1:172.21.15.146:6812/435598642 conn(0x7f3ee003e250 legacy=0x7f3ee0040710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 10 monclient: shutdown 2024-09-15T20:50:40.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:40.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 >> v1:172.21.15.78:6789/0 conn(0x7f3ef406e960 legacy=0x7f3ef411c5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 shutdown_connections 2024-09-15T20:50:40.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 >> 172.21.15.78:0/2386948336 conn(0x7f3ef40a63c0 msgr2=0x7f3ef40736c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:40.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 shutdown_connections 2024-09-15T20:50:40.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 1 -- 172.21.15.78:0/2386948336 wait complete. 2024-09-15T20:50:40.743 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.737+0000 7f3ef9cd3640 1 librados: shutdown 2024-09-15T20:50:40.758 INFO:tasks.ceph:Scrubbing osd.2 2024-09-15T20:50:40.758 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:50:40.881 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.877+0000 7fe697b33640 10 monclient: get_monmap_and_config 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.877+0000 7fe697b33640 10 monclient: build_initial_monmap 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.877+0000 7fe697b33640 10 monclient: monmap: 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:40.883865+0000 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:40.883865+0000 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:40.882 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:40.883 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.877+0000 7fe697b33640 1 Processor -- start 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.877+0000 7fe697b33640 1 -- start start 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.877+0000 7fe697b33640 10 monclient: init 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): picked mon.noname-a con 0x7fe690137fc0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): picked mon.noname-c con 0x7fe6900a0af0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): picked mon.noname-b con 0x7fe6900a4d70 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6900a6290 con 0x7fe690137fc0 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6900a1fa0 con 0x7fe6900a0af0 2024-09-15T20:50:40.890 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe6901383a0 con 0x7fe6900a4d70 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): authenticate will time out at 2301.768438s 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6958cf640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe690137fc0 0x7fe6900a84b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57894/0 (socket says 172.21.15.78:57894) 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6960d0640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe6900a4d70 0x7fe6900a92d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38486/0 (socket says 172.21.15.78:38486) 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6958cf640 1 -- 172.21.15.78:0/2276003912 learned_addr learned my addr 172.21.15.78:0/2276003912 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3814676836 0 0) 0x7fe6900a6290 con 0x7fe690137fc0 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient(hunting): my global_id is 144360 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe67c003530 con 0x7fe690137fc0 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4207310276 0 0) 0x7fe6901383a0 con 0x7fe6900a4d70 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.891 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient(hunting): my global_id is 154280 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe6900a6290 con 0x7fe6900a4d70 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3731160766 0 0) 0x7fe6900a6290 con 0x7fe6900a4d70 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 >> v1:172.21.15.146:6789/0 conn(0x7fe6900a0af0 legacy=0x7fe6900a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 >> v1:172.21.15.78:6789/0 conn(0x7fe690137fc0 legacy=0x7fe6900a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: _finish_hunting 0 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 monclient: found mon.noname-b 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe690138570 con 0x7fe6900a4d70 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: _finish_auth 0 2024-09-15T20:50:40.892 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: _check_auth_tickets 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe688001a00 con 0x7fe6900a4d70 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 5 monclient: authenticate success, global_id 154280 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: dump: 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.893 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe688001ba0 con 0x7fe6900a4d70 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 4 set_mon_vals no callback set 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 1 -- 172.21.15.78:0/2276003912 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe688001ff0 con 0x7fe6900a4d70 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6948cd640 10 monclient: dump: 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.894 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: shutdown 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- 172.21.15.78:0/2276003912 >> v1:172.21.15.78:6790/0 conn(0x7fe6900a4d70 legacy=0x7fe6900a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- 172.21.15.78:0/2276003912 shutdown_connections 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- 172.21.15.78:0/2276003912 >> 172.21.15.78:0/2276003912 conn(0x7fe6900aa420 msgr2=0x7fe6900aa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- 172.21.15.78:0/2276003912 shutdown_connections 2024-09-15T20:50:40.895 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- 172.21.15.78:0/2276003912 wait complete. 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: build_initial_monmap 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: monmap: 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.896 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 librados: starting msgr at 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 librados: starting objecter 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 Processor -- start 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- start start 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 librados: setting wanted keys 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 librados: calling monclient init 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: init 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): picked mon.noname-c con 0x7fe6900a0af0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): picked mon.noname-a con 0x7fe6900a4d70 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): picked mon.noname-b con 0x7fe690137fc0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe690135e80 con 0x7fe6900a4d70 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe690133040 con 0x7fe690137fc0 2024-09-15T20:50:40.897 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe690135ba0 con 0x7fe6900a0af0 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe697b33640 10 monclient(hunting): authenticate will time out at 2301.771399s 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6958cf640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fe6900a0af0 0x7fe690132220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38494/0 (socket says 172.21.15.78:38494) 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6950ce640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fe6900a4d70 0x7fe690132930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57896/0 (socket says 172.21.15.78:57896) 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6958cf640 1 -- 172.21.15.78:0/1477174193 learned_addr learned my addr 172.21.15.78:0/1477174193 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 client.?.objecter ms_handle_connect 0x7fe6900a0af0 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 client.?.objecter ms_handle_connect 0x7fe6900a4d70 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3950800909 0 0) 0x7fe690135ba0 con 0x7fe6900a0af0 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.898 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 monclient(hunting): my global_id is 154283 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe674004ec0 con 0x7fe6900a0af0 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2906091467 0 0) 0x7fe690135e80 con 0x7fe6900a4d70 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 monclient(hunting): my global_id is 144363 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe690135ba0 con 0x7fe6900a4d70 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 client.?.objecter ms_handle_connect 0x7fe690137fc0 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.881+0000 7fe6867fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3844504651 0 0) 0x7fe674004ec0 con 0x7fe6900a0af0 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe690135e80 con 0x7fe6900a0af0 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe68c0013a0 con 0x7fe6900a0af0 2024-09-15T20:50:40.899 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): dump: 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.900 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 774426645 0 0) 0x7fe690135ba0 con 0x7fe6900a4d70 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fe674004ec0 con 0x7fe6900a4d70 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe680001840 con 0x7fe6900a4d70 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): dump: 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.901 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3502942387 0 0) 0x7fe690133040 con 0x7fe690137fc0 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient(hunting): my global_id is 164179 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe690135ba0 con 0x7fe690137fc0 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 877795295 0 0) 0x7fe690135e80 con 0x7fe6900a0af0 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 >> v1:172.21.15.146:6789/0 conn(0x7fe690137fc0 legacy=0x7fe6901d4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 >> v1:172.21.15.78:6789/0 conn(0x7fe6900a4d70 legacy=0x7fe690132930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.902 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: _finish_hunting 0 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 monclient: found mon.c 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe690134680 con 0x7fe6900a0af0 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: _finish_auth 0 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: _check_auth_tickets 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 5 monclient: authenticate success, global_id 154283 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 10 monclient: _renew_subs 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fe690137570 con 0x7fe6900a0af0 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 10 client.154283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 10 monclient: _renew_subs 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.903 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fe6901d66c0 con 0x7fe6900a0af0 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 1 librados: init done 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fe68c001600 con 0x7fe6900a0af0 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fe68c004c40 con 0x7fe6900a0af0 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: dump: 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:40.904 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe687fff640 4 set_mon_vals no callback set 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 10 client.154283.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 15 client.154283.objecter _session_command_op_assign -1 1 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 20 client.154283.objecter _get_session osd=-1 returning homeless 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 10 client.154283.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fe68c010cb0 con 0x7fe6900a0af0 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fe678002920 con 0x7fe6900a0af0 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fe68c04b550 con 0x7fe6900a0af0 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 client.154283.objecter ms_dispatch 0x7fe690000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:40.905 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 3 client.154283.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 3 client.154283.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 client.154283.objecter checking command 1 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 client.154283.objecter _get_session s=0x7fe6740460c0 osd=2 3 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 client.154283.objecter put_session s=0x7fe6740460c0 osd=2 3 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 15 client.154283.objecter _session_command_op_remove -1 1 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 client.154283.objecter _get_session s=0x7fe6740460c0 osd=2 3 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 client.154283.objecter get_session s=0x7fe6740460c0 osd=2 3 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 15 client.154283.objecter _session_command_op_assign 2 1 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 client.154283.objecter put_session s=0x7fe6740460c0 osd=2 4 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 client.154283.objecter _send_command 1 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6808/404661851 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe67404a480 con 0x7fe674046770 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 client.154283.objecter dump_active .. 0 homeless 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 3222990830 0 0) 0x7fe68c04b960 con 0x7fe6900a0af0 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-15T20:50:40.906 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6960d0640 10 monclient: get_auth_request con 0x7fe674046770 auth_method 0 2024-09-15T20:50:40.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6950ce640 10 monclient: get_auth_request con 0x7fe67403e240 auth_method 0 2024-09-15T20:50:40.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 client.154283.objecter ms_handle_connect 0x7fe674046770 2024-09-15T20:50:40.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 client.154283.objecter ms_handle_connect 0x7fe67403e240 2024-09-15T20:50:40.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== osd.2 v1:172.21.15.78:6808/404661851 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fe67404a480 con 0x7fe674046770 2024-09-15T20:50:40.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 client.154283.objecter ms_dispatch 0x7fe690000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:50:40.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 10 client.154283.objecter _finish_command 1 = system:0 2024-09-15T20:50:40.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 20 client.154283.objecter put_session s=0x7fe6740460c0 osd=2 4 2024-09-15T20:50:40.907 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.885+0000 7fe6867fc640 15 client.154283.objecter _session_command_op_remove 2 1 2024-09-15T20:50:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 10 client.154283.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T20:50:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 15 client.154283.objecter _session_command_op_assign -1 2 2024-09-15T20:50:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 20 client.154283.objecter _get_session s=0x7fe6740460c0 osd=2 3 2024-09-15T20:50:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 20 client.154283.objecter put_session s=0x7fe6740460c0 osd=2 3 2024-09-15T20:50:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 20 client.154283.objecter _get_session s=0x7fe6740460c0 osd=2 3 2024-09-15T20:50:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 15 client.154283.objecter _session_command_op_remove -1 2 2024-09-15T20:50:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 20 client.154283.objecter get_session s=0x7fe6740460c0 osd=2 3 2024-09-15T20:50:40.908 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 15 client.154283.objecter _session_command_op_assign 2 2 2024-09-15T20:50:40.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 20 client.154283.objecter put_session s=0x7fe6740460c0 osd=2 4 2024-09-15T20:50:40.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 10 client.154283.objecter _send_command 2 2024-09-15T20:50:40.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 --> v1:172.21.15.78:6808/404661851 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fe6780012c0 con 0x7fe674046770 2024-09-15T20:50:40.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe6867fc640 1 -- 172.21.15.78:0/1477174193 <== osd.2 v1:172.21.15.78:6808/404661851 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fe6780012c0 con 0x7fe674046770 2024-09-15T20:50:40.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe6867fc640 10 client.154283.objecter ms_dispatch 0x7fe690000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:50:40.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe6867fc640 10 client.154283.objecter _finish_command 2 = system:0 2024-09-15T20:50:40.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe6867fc640 20 client.154283.objecter put_session s=0x7fe6740460c0 osd=2 4 2024-09-15T20:50:40.909 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe6867fc640 15 client.154283.objecter _session_command_op_remove 2 2 2024-09-15T20:50:40.910 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:40.910 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:50:40.910 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 10 librados: watch_flush enter 2024-09-15T20:50:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 10 librados: watch_flush exit 2024-09-15T20:50:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 10 client.154283.objecter close_session for osd.2 2024-09-15T20:50:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 >> v1:172.21.15.78:6808/404661851 conn(0x7fe674046770 legacy=0x7fe674048c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 20 client.154283.objecter put_session s=0x7fe6740460c0 osd=2 1 2024-09-15T20:50:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 20 client.154283.objecter shutdown clearing up homeless session... 2024-09-15T20:50:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 10 client.154283.objecter successfully canceled tick 2024-09-15T20:50:40.911 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 >> v1:172.21.15.146:6812/435598642 conn(0x7fe67403e240 legacy=0x7fe674040700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 10 monclient: shutdown 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 >> v1:172.21.15.78:6790/0 conn(0x7fe6900a0af0 legacy=0x7fe690132220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe6960d0640 1 -- 172.21.15.78:0/1477174193 reap_dead start 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 shutdown_connections 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.897+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 >> 172.21.15.78:0/1477174193 conn(0x7fe6900aa420 msgr2=0x7fe6900a2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.901+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 shutdown_connections 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.901+0000 7fe697b33640 1 -- 172.21.15.78:0/1477174193 wait complete. 2024-09-15T20:50:40.912 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:40.901+0000 7fe697b33640 1 librados: shutdown 2024-09-15T20:50:40.917 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-09-15T20:50:41.033 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.029+0000 7ff60107c640 10 monclient: get_monmap_and_config 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.029+0000 7ff60107c640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.029+0000 7ff60107c640 10 monclient: monmap: 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:41.035964+0000 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:41.035964+0000 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:41.034 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:41.035 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.029+0000 7ff60107c640 1 Processor -- start 2024-09-15T20:50:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.029+0000 7ff60107c640 1 -- start start 2024-09-15T20:50:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.029+0000 7ff60107c640 10 monclient: init 2024-09-15T20:50:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:50:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): picked mon.noname-c con 0x7ff5fc135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): picked mon.noname-a con 0x7ff5fc0a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.041 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): picked mon.noname-b con 0x7ff5fc13c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc0745e0 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc06ecf0 con 0x7ff5fc135b60 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc06efd0 con 0x7ff5fc13c220 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): authenticate will time out at 2301.920530s 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5fa59c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7ff5fc0a24e0 0x7ff5fc13ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57910/0 (socket says 172.21.15.78:57910) 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5fb59e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff5fc13c220 0x7ff5fc13e610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38508/0 (socket says 172.21.15.78:38508) 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5fa59c640 1 -- 172.21.15.78:0/719280326 learned_addr learned my addr 172.21.15.78:0/719280326 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3012341629 0 0) 0x7ff5fc0745e0 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient(hunting): my global_id is 144366 2024-09-15T20:50:41.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e4003530 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2813438163 0 0) 0x7ff5fc06efd0 con 0x7ff5fc13c220 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient(hunting): my global_id is 154286 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5fc0745e0 con 0x7ff5fc13c220 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1761980494 0 0) 0x7ff5e4003530 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 >> v1:172.21.15.78:6790/0 conn(0x7ff5fc13c220 legacy=0x7ff5fc13e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 >> v1:172.21.15.146:6789/0 conn(0x7ff5fc135b60 legacy=0x7ff5fc135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 monclient: found mon.noname-a 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.043 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc136650 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff5f00016a0 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: dump: 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.044 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 5 monclient: authenticate success, global_id 144366 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5f0001840 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 1 -- 172.21.15.78:0/719280326 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff5f0001c10 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5f9d9b640 10 monclient: dump: 2024-09-15T20:50:41.045 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 4 set_mon_vals no callback set 2024-09-15T20:50:41.046 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: shutdown 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- 172.21.15.78:0/719280326 >> v1:172.21.15.78:6789/0 conn(0x7ff5fc0a24e0 legacy=0x7ff5fc13ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- 172.21.15.78:0/719280326 shutdown_connections 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- 172.21.15.78:0/719280326 >> 172.21.15.78:0/719280326 conn(0x7ff5fc0a63c0 msgr2=0x7ff5fc0a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- 172.21.15.78:0/719280326 shutdown_connections 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- 172.21.15.78:0/719280326 wait complete. 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: monmap: 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:41.047 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 librados: starting msgr at 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 librados: starting objecter 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 Processor -- start 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- start start 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 librados: setting wanted keys 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 librados: calling monclient init 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: init 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.048 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): picked mon.noname-c con 0x7ff5fc0a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): picked mon.noname-b con 0x7ff5fc135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): picked mon.noname-a con 0x7ff5fc13c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc09d180 con 0x7ff5fc13c220 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc1393d0 con 0x7ff5fc135b60 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5fc09ef80 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff60107c640 10 monclient(hunting): authenticate will time out at 2301.923752s 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5fad9d640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7ff5fc0a24e0 0x7ff5fc1cce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38510/0 (socket says 172.21.15.78:38510) 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5fad9d640 1 -- 172.21.15.78:0/1498410857 learned_addr learned my addr 172.21.15.78:0/1498410857 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5fb59e640 1 --1- 172.21.15.78:0/1498410857 >> v1:172.21.15.78:6789/0 conn(0x7ff5fc13c220 0x7ff5fc1d55c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57916/0 (socket says 172.21.15.78:57916) 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5e37fe640 10 client.?.objecter ms_handle_connect 0x7ff5fc13c220 2024-09-15T20:50:41.049 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5e37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5e37fe640 10 client.?.objecter ms_handle_connect 0x7ff5fc0a24e0 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.033+0000 7ff5e37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2877835982 0 0) 0x7ff5fc09ef80 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): my global_id is 154289 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e8004ec0 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2806590436 0 0) 0x7ff5fc0745e0 con 0x7ff5fc13c220 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.050 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): my global_id is 144369 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5fc09ef80 con 0x7ff5fc13c220 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 client.?.objecter ms_handle_connect 0x7ff5fc135b60 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2771296937 0 0) 0x7ff5fc09ef80 con 0x7ff5fc13c220 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5fc0745e0 con 0x7ff5fc13c220 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 502116549 0 0) 0x7ff5e8004ec0 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7ff5fc09ef80 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff5ec001bf0 con 0x7ff5fc13c220 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): dump: 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.051 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff5f4001940 con 0x7ff5fc0a24e0 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): dump: 2024-09-15T20:50:41.052 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2169271479 0 0) 0x7ff5fc136650 con 0x7ff5fc135b60 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.053 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient(hunting): my global_id is 164185 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e8004ec0 con 0x7ff5fc135b60 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1862315797 0 0) 0x7ff5fc0745e0 con 0x7ff5fc13c220 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 >> v1:172.21.15.78:6790/0 conn(0x7ff5fc0a24e0 legacy=0x7ff5fc1cce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 >> v1:172.21.15.146:6789/0 conn(0x7ff5fc135b60 legacy=0x7ff5fc1d1e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 monclient: found mon.a 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5fc139c70 con 0x7ff5fc13c220 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.054 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 5 monclient: authenticate success, global_id 144369 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 10 monclient: _renew_subs 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7ff5fc135590 con 0x7ff5fc13c220 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 10 client.144369.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 10 monclient: _renew_subs 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7ff5fc09e4d0 con 0x7ff5fc13c220 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 1 librados: init done 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7ff5ec004b70 con 0x7ff5fc13c220 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7ff5ec004ec0 con 0x7ff5fc13c220 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.055 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5f959a640 4 set_mon_vals no callback set 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 monclient: dump: 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.056 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5dc005090 con 0x7ff5fc13c220 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7ff5ec010f30 con 0x7ff5fc13c220 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7ff5ec04b750 con 0x7ff5fc13c220 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 client.144369.objecter ms_dispatch 0x7ff5fc000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 3 client.144369.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 3 client.144369.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 20 client.144369.objecter dump_active .. 0 homeless 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5fad9d640 10 monclient: get_auth_request con 0x7ff5e803e220 auth_method 0 2024-09-15T20:50:41.057 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.037+0000 7ff5e37fe640 10 client.144369.objecter ms_handle_connect 0x7ff5e803e220 2024-09-15T20:50:41.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.041+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7ff5fc09d180 con 0x7ff5fc13c220 2024-09-15T20:50:41.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.041+0000 7ff5e37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.041+0000 7ff5e37fe640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:41.214 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.209+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7ff5dc000ba0 con 0x7ff5e803e220 2024-09-15T20:50:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff5e37fe640 1 -- 172.21.15.78:0/1498410857 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (unknown 3049115190 0 0) 0x7ff5dc000ba0 con 0x7ff5e803e220 2024-09-15T20:50:41.215 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 2 to deep-scrub 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 10 librados: watch_flush enter 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 10 librados: watch_flush exit 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 20 client.144369.objecter shutdown clearing up homeless session... 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 10 client.144369.objecter successfully canceled tick 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 >> v1:172.21.15.146:6812/435598642 conn(0x7ff5e803e220 legacy=0x7ff5e80406e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 10 monclient: shutdown 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 >> v1:172.21.15.78:6789/0 conn(0x7ff5fc13c220 legacy=0x7ff5fc1d55c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 shutdown_connections 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 >> 172.21.15.78:0/1498410857 conn(0x7ff5fc0a63c0 msgr2=0x7ff5fc13f060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 shutdown_connections 2024-09-15T20:50:41.218 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 1 -- 172.21.15.78:0/1498410857 wait complete. 2024-09-15T20:50:41.219 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.213+0000 7ff60107c640 1 librados: shutdown 2024-09-15T20:50:41.233 INFO:tasks.ceph:Scrubbing osd.3 2024-09-15T20:50:41.234 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: get_monmap_and_config 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: monmap: 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:41.357266+0000 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:41.357266+0000 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.355 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:41.356 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:41.356 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 Processor -- start 2024-09-15T20:50:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- start start 2024-09-15T20:50:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: init 2024-09-15T20:50:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:50:41.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient(hunting): picked mon.noname-c con 0x7f1c441378c0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient(hunting): picked mon.noname-a con 0x7f1c441383b0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient(hunting): picked mon.noname-b con 0x7f1c4413e580 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4409fbe0 con 0x7f1c441383b0 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c44072850 con 0x7f1c441378c0 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c44073d70 con 0x7f1c4413e580 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient(hunting): authenticate will time out at 2302.241824s 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c4359e640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f1c4413e580 0x7f1c44140970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38514/0 (socket says 172.21.15.78:38514) 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c4359e640 1 -- 172.21.15.78:0/589365618 learned_addr learned my addr 172.21.15.78:0/589365618 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c4259c640 1 --1- 172.21.15.78:0/589365618 >> v1:172.21.15.78:6789/0 conn(0x7f1c441383b0 0x7f1c4413ce60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57930/0 (socket says 172.21.15.78:57930) 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2067680016 0 0) 0x7f1c44073d70 con 0x7f1c4413e580 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient(hunting): my global_id is 154292 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c2c003530 con 0x7f1c4413e580 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1509223086 0 0) 0x7f1c4409fbe0 con 0x7f1c441383b0 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient(hunting): my global_id is 144372 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c44073d70 con 0x7f1c441383b0 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2176553433 0 0) 0x7f1c2c003530 con 0x7f1c4413e580 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 >> v1:172.21.15.146:6789/0 conn(0x7f1c441378c0 legacy=0x7f1c44137ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 >> v1:172.21.15.78:6789/0 conn(0x7f1c441383b0 legacy=0x7f1c4413ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 monclient: found mon.noname-b 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c44074b10 con 0x7f1c4413e580 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1c3c001a20 con 0x7f1c4413e580 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: dump: 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.367 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 5 monclient: authenticate success, global_id 154292 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c3c001bc0 con 0x7f1c4413e580 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 1 -- 172.21.15.78:0/589365618 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1c3c002050 con 0x7f1c4413e580 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 4 set_mon_vals no callback set 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c41d9b640 10 monclient: dump: 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: shutdown 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- 172.21.15.78:0/589365618 >> v1:172.21.15.78:6790/0 conn(0x7f1c4413e580 legacy=0x7f1c44140970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- 172.21.15.78:0/589365618 shutdown_connections 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- 172.21.15.78:0/589365618 >> 172.21.15.78:0/589365618 conn(0x7f1c440a6240 msgr2=0x7f1c440a6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- 172.21.15.78:0/589365618 shutdown_connections 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 1 -- 172.21.15.78:0/589365618 wait complete. 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.353+0000 7f1c49203640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: monmap: 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:41.370 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 librados: starting msgr at 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 librados: starting objecter 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 Processor -- start 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 -- start start 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 librados: setting wanted keys 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 librados: calling monclient init 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: init 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient(hunting): picked mon.noname-b con 0x7f1c441378c0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient(hunting): picked mon.noname-a con 0x7f1c441383b0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient(hunting): picked mon.noname-c con 0x7f1c4413e580 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4411ccc0 con 0x7f1c441383b0 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4413b6b0 con 0x7f1c441378c0 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1c4413de90 con 0x7f1c4413e580 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient(hunting): authenticate will time out at 2302.244929s 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c4259c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f1c441383b0 0x7f1c4413afa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57940/0 (socket says 172.21.15.78:57940) 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c4259c640 1 -- 172.21.15.78:0/4131671683 learned_addr learned my addr 172.21.15.78:0/4131671683 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.?.objecter ms_handle_connect 0x7f1c4413e580 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.?.objecter ms_handle_connect 0x7f1c441383b0 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1454084682 0 0) 0x7f1c4413de90 con 0x7f1c4413e580 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): my global_id is 154295 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c30004ec0 con 0x7f1c4413e580 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2031832246 0 0) 0x7f1c4411ccc0 con 0x7f1c441383b0 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): my global_id is 144375 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c4413de90 con 0x7f1c441383b0 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.?.objecter ms_handle_connect 0x7f1c441378c0 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 919120725 0 0) 0x7f1c30004ec0 con 0x7f1c4413e580 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c4411ccc0 con 0x7f1c4413e580 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1c3c001f10 con 0x7f1c4413e580 2024-09-15T20:50:41.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): dump: 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:41.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3026010224 0 0) 0x7f1c4413de90 con 0x7f1c441383b0 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f1c30004ec0 con 0x7f1c441383b0 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1c34001360 con 0x7f1c441383b0 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): dump: 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.375 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 596064681 0 0) 0x7f1c4413b6b0 con 0x7f1c441378c0 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient(hunting): my global_id is 164191 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1c4413de90 con 0x7f1c441378c0 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3219546772 0 0) 0x7f1c4411ccc0 con 0x7f1c4413e580 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 >> v1:172.21.15.146:6789/0 conn(0x7f1c441378c0 legacy=0x7f1c4413a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 >> v1:172.21.15.78:6789/0 conn(0x7f1c441383b0 legacy=0x7f1c4413afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 monclient: found mon.c 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.376 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c4413e350 con 0x7f1c4413e580 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 5 monclient: authenticate success, global_id 154295 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: _renew_subs 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f1c4409d0a0 con 0x7f1c4413e580 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 client.154295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: _renew_subs 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f1c4413d110 con 0x7f1c4413e580 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 librados: init done 2024-09-15T20:50:41.377 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f1c3c0020b0 con 0x7f1c4413e580 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f1c3c004f80 con 0x7f1c4413e580 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c4159a640 4 set_mon_vals no callback set 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: dump: 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.378 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 client.154295.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f1c3c010ff0 con 0x7f1c4413e580 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 15 client.154295.objecter _session_command_op_assign -1 1 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 20 client.154295.objecter _get_session osd=-1 returning homeless 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 client.154295.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f1c24002920 con 0x7f1c4413e580 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f1c3c04b6e0 con 0x7f1c4413e580 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.154295.objecter ms_dispatch 0x7f1c44000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 3 client.154295.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 3 client.154295.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.154295.objecter checking command 1 2024-09-15T20:50:41.379 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 20 client.154295.objecter _get_session s=0x7f1c30046140 osd=3 3 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 20 client.154295.objecter put_session s=0x7f1c30046140 osd=3 3 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 15 client.154295.objecter _session_command_op_remove -1 1 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 20 client.154295.objecter _get_session s=0x7f1c30046140 osd=3 3 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 20 client.154295.objecter get_session s=0x7f1c30046140 osd=3 3 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 15 client.154295.objecter _session_command_op_assign 3 1 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 20 client.154295.objecter put_session s=0x7f1c30046140 osd=3 4 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.154295.objecter _send_command 1 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.146:6800/490126554 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1c3004a500 con 0x7f1c300467f0 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 20 client.154295.objecter dump_active .. 0 homeless 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 3222990830 0 0) 0x7f1c3c04ba70 con 0x7f1c4413e580 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c42d9d640 10 monclient: get_auth_request con 0x7f1c3003e2b0 auth_method 0 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c4259c640 10 monclient: get_auth_request con 0x7f1c300467f0 auth_method 0 2024-09-15T20:50:41.380 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.154295.objecter ms_handle_connect 0x7f1c3003e2b0 2024-09-15T20:50:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.357+0000 7f1c2b7fe640 10 client.154295.objecter ms_handle_connect 0x7f1c300467f0 2024-09-15T20:50:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.361+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== osd.3 v1:172.21.15.146:6800/490126554 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f1c3004a500 con 0x7f1c300467f0 2024-09-15T20:50:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.361+0000 7f1c2b7fe640 10 client.154295.objecter ms_dispatch 0x7f1c44000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:50:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.361+0000 7f1c2b7fe640 10 client.154295.objecter _finish_command 1 = system:0 2024-09-15T20:50:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.361+0000 7f1c2b7fe640 20 client.154295.objecter put_session s=0x7f1c30046140 osd=3 4 2024-09-15T20:50:41.381 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.361+0000 7f1c2b7fe640 15 client.154295.objecter _session_command_op_remove 3 1 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 10 client.154295.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 15 client.154295.objecter _session_command_op_assign -1 2 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 20 client.154295.objecter _get_session s=0x7f1c30046140 osd=3 3 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 20 client.154295.objecter put_session s=0x7f1c30046140 osd=3 3 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 20 client.154295.objecter _get_session s=0x7f1c30046140 osd=3 3 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 15 client.154295.objecter _session_command_op_remove -1 2 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 20 client.154295.objecter get_session s=0x7f1c30046140 osd=3 3 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 15 client.154295.objecter _session_command_op_assign 3 2 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 20 client.154295.objecter put_session s=0x7f1c30046140 osd=3 4 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 10 client.154295.objecter _send_command 2 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 --> v1:172.21.15.146:6800/490126554 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1c240012c0 con 0x7f1c300467f0 2024-09-15T20:50:41.382 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c2b7fe640 1 -- 172.21.15.78:0/4131671683 <== osd.3 v1:172.21.15.146:6800/490126554 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f1c240012c0 con 0x7f1c300467f0 2024-09-15T20:50:41.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c2b7fe640 10 client.154295.objecter ms_dispatch 0x7f1c44000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:50:41.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c2b7fe640 10 client.154295.objecter _finish_command 2 = system:0 2024-09-15T20:50:41.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c2b7fe640 20 client.154295.objecter put_session s=0x7f1c30046140 osd=3 4 2024-09-15T20:50:41.383 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c2b7fe640 15 client.154295.objecter _session_command_op_remove 3 2 2024-09-15T20:50:41.383 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:41.383 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:50:41.383 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:41.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 10 librados: watch_flush enter 2024-09-15T20:50:41.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 10 librados: watch_flush exit 2024-09-15T20:50:41.384 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 10 client.154295.objecter close_session for osd.3 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 >> v1:172.21.15.146:6800/490126554 conn(0x7f1c300467f0 legacy=0x7f1c30048c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 20 client.154295.objecter put_session s=0x7f1c30046140 osd=3 1 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 20 client.154295.objecter shutdown clearing up homeless session... 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 10 client.154295.objecter successfully canceled tick 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 >> v1:172.21.15.146:6812/435598642 conn(0x7f1c3003e2b0 legacy=0x7f1c30040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 10 monclient: shutdown 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 >> v1:172.21.15.78:6790/0 conn(0x7f1c4413e580 legacy=0x7f1c441d5060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c4359e640 1 -- 172.21.15.78:0/4131671683 reap_dead start 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 shutdown_connections 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.373+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 >> 172.21.15.78:0/4131671683 conn(0x7f1c440a6240 msgr2=0x7f1c44137100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.377+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 shutdown_connections 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.377+0000 7f1c49203640 1 -- 172.21.15.78:0/4131671683 wait complete. 2024-09-15T20:50:41.385 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.377+0000 7f1c49203640 1 librados: shutdown 2024-09-15T20:50:41.395 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-09-15T20:50:41.510 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.505+0000 7fa7add62640 10 monclient: get_monmap_and_config 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.505+0000 7fa7add62640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: monmap: 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:41.512920+0000 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:41.512920+0000 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:41.511 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.517 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 Processor -- start 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- start start 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: init 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a8134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a809f5a0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a806e960 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a80a0470 con 0x7fa7a8134bb0 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a809feb0 con 0x7fa7a809f5a0 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a809e4a0 con 0x7fa7a806e960 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient(hunting): authenticate will time out at 2302.397352s 2024-09-15T20:50:41.518 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a77fe640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa7a8134bb0 0x7fa7a8134f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57942/0 (socket says 172.21.15.78:57942) 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a7fff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa7a806e960 0x7fa7a813d0f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38536/0 (socket says 172.21.15.78:38536) 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a77fe640 1 -- 172.21.15.78:0/4104550192 learned_addr learned my addr 172.21.15.78:0/4104550192 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2928109368 0 0) 0x7fa798001120 con 0x7fa7a8134bb0 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient(hunting): my global_id is 144378 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa790003530 con 0x7fa7a8134bb0 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 203350950 0 0) 0x7fa7a809e4a0 con 0x7fa7a806e960 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient(hunting): my global_id is 154298 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa798001120 con 0x7fa7a806e960 2024-09-15T20:50:41.519 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 250392735 0 0) 0x7fa790003530 con 0x7fa7a8134bb0 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 >> v1:172.21.15.78:6790/0 conn(0x7fa7a806e960 legacy=0x7fa7a813d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 >> v1:172.21.15.146:6789/0 conn(0x7fa7a809f5a0 legacy=0x7fa7a81399c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 monclient: found mon.noname-a 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a80a0ff0 con 0x7fa7a8134bb0 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa798001870 con 0x7fa7a8134bb0 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 5 monclient: authenticate success, global_id 144378 2024-09-15T20:50:41.520 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: dump: 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:41.521 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa798001a90 con 0x7fa7a8134bb0 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 1 -- 172.21.15.78:0/4104550192 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa798004a50 con 0x7fa7a8134bb0 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 4 set_mon_vals no callback set 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7a67fc640 10 monclient: dump: 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.522 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: shutdown 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- 172.21.15.78:0/4104550192 >> v1:172.21.15.78:6789/0 conn(0x7fa7a8134bb0 legacy=0x7fa7a8134f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- 172.21.15.78:0/4104550192 shutdown_connections 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- 172.21.15.78:0/4104550192 >> 172.21.15.78:0/4104550192 conn(0x7fa7a80a63c0 msgr2=0x7fa7a80a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- 172.21.15.78:0/4104550192 shutdown_connections 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- 172.21.15.78:0/4104550192 wait complete. 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: monmap: 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.523 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 librados: starting msgr at 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 librados: starting objecter 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 Processor -- start 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 -- start start 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 librados: setting wanted keys 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 1 librados: calling monclient init 2024-09-15T20:50:41.524 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.509+0000 7fa7add62640 10 monclient: init 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient(hunting): picked mon.noname-b con 0x7fa7a806e960 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient(hunting): picked mon.noname-c con 0x7fa7a809f5a0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient(hunting): picked mon.noname-a con 0x7fa7a8134bb0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a813d4c0 con 0x7fa7a8134bb0 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a813d7a0 con 0x7fa7a806e960 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa7a80a5c10 con 0x7fa7a809f5a0 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7a7fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fa7a8134bb0 0x7fa7a81d2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57956/0 (socket says 172.21.15.78:57956) 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7a7fff640 1 -- 172.21.15.78:0/2726067736 learned_addr learned my addr 172.21.15.78:0/2726067736 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7a6ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fa7a809f5a0 0x7fa7a81ced50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38544/0 (socket says 172.21.15.78:38544) 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.525 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient(hunting): authenticate will time out at 2302.400511s 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 client.?.objecter ms_handle_connect 0x7fa7a8134bb0 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 client.?.objecter ms_handle_connect 0x7fa7a809f5a0 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2514041671 0 0) 0x7fa7a80a5c10 con 0x7fa7a809f5a0 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): my global_id is 154301 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7940070d0 con 0x7fa7a809f5a0 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 20899537 0 0) 0x7fa7a813d4c0 con 0x7fa7a8134bb0 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.526 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): my global_id is 144381 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a80a5c10 con 0x7fa7a8134bb0 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 client.?.objecter ms_handle_connect 0x7fa7a806e960 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2164579885 0 0) 0x7fa7940070d0 con 0x7fa7a809f5a0 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7a813d4c0 con 0x7fa7a809f5a0 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa79c001980 con 0x7fa7a809f5a0 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): dump: 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.527 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1237036286 0 0) 0x7fa7a80a5c10 con 0x7fa7a8134bb0 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fa7940070d0 con 0x7fa7a8134bb0 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4155803166 0 0) 0x7fa7a813d7a0 con 0x7fa7a806e960 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.528 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): my global_id is 164197 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa7a80a5c10 con 0x7fa7a806e960 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa7a0001620 con 0x7fa7a8134bb0 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient(hunting): dump: 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.529 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1534438213 0 0) 0x7fa7a813d4c0 con 0x7fa7a809f5a0 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 >> v1:172.21.15.146:6789/0 conn(0x7fa7a806e960 legacy=0x7fa7a81c9d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 >> v1:172.21.15.78:6789/0 conn(0x7fa7a8134bb0 legacy=0x7fa7a81d2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 monclient: found mon.c 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a80a5360 con 0x7fa7a809f5a0 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.530 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 5 monclient: authenticate success, global_id 154301 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient: _renew_subs 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a809d180 con 0x7fa7a809f5a0 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 client.154301.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient: _renew_subs 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fa7a813ae30 con 0x7fa7a809f5a0 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 1 librados: init done 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fa79c001c80 con 0x7fa7a809f5a0 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fa79c003a70 con 0x7fa7a809f5a0 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.531 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 monclient: dump: 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7a5ffb640 4 set_mon_vals no callback set 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.532 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa788005090 con 0x7fa7a809f5a0 2024-09-15T20:50:41.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fa79c003c10 con 0x7fa7a809f5a0 2024-09-15T20:50:41.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fa79c04b550 con 0x7fa7a809f5a0 2024-09-15T20:50:41.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 client.154301.objecter ms_dispatch 0x7fa7a8000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:41.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 3 client.154301.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:41.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 3 client.154301.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:41.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 20 client.154301.objecter dump_active .. 0 homeless 2024-09-15T20:50:41.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa7a77fe640 10 monclient: get_auth_request con 0x7fa7940404c0 auth_method 0 2024-09-15T20:50:41.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.513+0000 7fa78ffff640 10 client.154301.objecter ms_handle_connect 0x7fa7940404c0 2024-09-15T20:50:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.517+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fa79c016ab0 con 0x7fa7a809f5a0 2024-09-15T20:50:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.517+0000 7fa78ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.517+0000 7fa78ffff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:41.680 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.677+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fa788000ba0 con 0x7fa7940404c0 2024-09-15T20:50:41.681 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.677+0000 7fa78ffff640 1 -- 172.21.15.78:0/2726067736 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (unknown 3526570807 0 0) 0x7fa788000ba0 con 0x7fa7940404c0 2024-09-15T20:50:41.681 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 3 to deep-scrub 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 10 librados: watch_flush enter 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 10 librados: watch_flush exit 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 20 client.154301.objecter shutdown clearing up homeless session... 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 10 client.154301.objecter successfully canceled tick 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 >> v1:172.21.15.146:6812/435598642 conn(0x7fa7940404c0 legacy=0x7fa794042980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 10 monclient: shutdown 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 >> v1:172.21.15.78:6790/0 conn(0x7fa7a809f5a0 legacy=0x7fa7a81ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 shutdown_connections 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 >> 172.21.15.78:0/2726067736 conn(0x7fa7a80a63c0 msgr2=0x7fa7a806f250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 shutdown_connections 2024-09-15T20:50:41.684 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 1 -- 172.21.15.78:0/2726067736 wait complete. 2024-09-15T20:50:41.685 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.681+0000 7fa7add62640 1 librados: shutdown 2024-09-15T20:50:41.698 INFO:tasks.ceph:Scrubbing osd.4 2024-09-15T20:50:41.698 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: get_monmap_and_config 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: monmap: 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:41.813246+0000 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:41.813246+0000 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.811 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.812 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:41.812 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:41.812 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.814 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 Processor -- start 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- start start 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: init 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient(hunting): picked mon.noname-a con 0x7fcad00a1300 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient(hunting): picked mon.noname-c con 0x7fcad01341b0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient(hunting): picked mon.noname-b con 0x7fcad013a790 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad00a0290 con 0x7fcad00a1300 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad009f900 con 0x7fcad01341b0 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad0134690 con 0x7fcad013a790 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient(hunting): authenticate will time out at 2302.697865s 2024-09-15T20:50:41.815 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcaceffd640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcad00a1300 0x7fcad00742a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57970/0 (socket says 172.21.15.78:57970) 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcaceffd640 1 -- 172.21.15.78:0/1451058723 learned_addr learned my addr 172.21.15.78:0/1451058723 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1559614296 0 0) 0x7fcad0134690 con 0x7fcad013a790 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient(hunting): my global_id is 154304 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcab8003530 con 0x7fcad013a790 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3115421794 0 0) 0x7fcad00a0290 con 0x7fcad00a1300 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient(hunting): my global_id is 144384 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcad0134690 con 0x7fcad00a1300 2024-09-15T20:50:41.816 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2876634371 0 0) 0x7fcab8003530 con 0x7fcad013a790 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 >> v1:172.21.15.146:6789/0 conn(0x7fcad01341b0 legacy=0x7fcad0139070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 >> v1:172.21.15.78:6789/0 conn(0x7fcad00a1300 legacy=0x7fcad00742a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 monclient: found mon.noname-b 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad009f340 con 0x7fcad013a790 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcac8001a40 con 0x7fcad013a790 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 5 monclient: authenticate success, global_id 154304 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:41.817 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: dump: 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcac8001be0 con 0x7fcad013a790 2024-09-15T20:50:41.818 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 1 -- 172.21.15.78:0/1451058723 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcac8001fb0 con 0x7fcad013a790 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 4 set_mon_vals no callback set 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcacdffb640 10 monclient: dump: 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.819 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: shutdown 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- 172.21.15.78:0/1451058723 >> v1:172.21.15.78:6790/0 conn(0x7fcad013a790 legacy=0x7fcad013cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- 172.21.15.78:0/1451058723 shutdown_connections 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- 172.21.15.78:0/1451058723 >> 172.21.15.78:0/1451058723 conn(0x7fcad00a6240 msgr2=0x7fcad00a6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- 172.21.15.78:0/1451058723 shutdown_connections 2024-09-15T20:50:41.820 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 1 -- 172.21.15.78:0/1451058723 wait complete. 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.809+0000 7fcad5779640 10 monclient: monmap: 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.821 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 librados: starting msgr at 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 librados: starting objecter 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 Processor -- start 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 -- start start 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 librados: setting wanted keys 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 librados: calling monclient init 2024-09-15T20:50:41.822 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient: init 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient(hunting): picked mon.noname-c con 0x7fcad00a1300 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient(hunting): picked mon.noname-a con 0x7fcad01341b0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient(hunting): picked mon.noname-b con 0x7fcad013a790 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad0070cb0 con 0x7fcad01341b0 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad006ef60 con 0x7fcad013a790 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcad0074550 con 0x7fcad00a1300 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient(hunting): authenticate will time out at 2302.700951s 2024-09-15T20:50:41.827 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcaceffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fcad00a1300 0x7fcad006e140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38562/0 (socket says 172.21.15.78:38562) 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcaceffd640 1 -- 172.21.15.78:0/869983919 learned_addr learned my addr 172.21.15.78:0/869983919 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcace7fc640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fcad01341b0 0x7fcad006e850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57976/0 (socket says 172.21.15.78:57976) 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.?.objecter ms_handle_connect 0x7fcad00a1300 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.?.objecter ms_handle_connect 0x7fcad01341b0 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3116749792 0 0) 0x7fcad0074550 con 0x7fcad00a1300 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): my global_id is 154307 2024-09-15T20:50:41.828 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcabc004ec0 con 0x7fcad00a1300 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 706445901 0 0) 0x7fcad0070cb0 con 0x7fcad01341b0 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): my global_id is 144387 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcad0074550 con 0x7fcad01341b0 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.?.objecter ms_handle_connect 0x7fcad013a790 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 1639784318 0 0) 0x7fcad0074550 con 0x7fcad01341b0 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcad0070cb0 con 0x7fcad01341b0 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3277347301 0 0) 0x7fcabc004ec0 con 0x7fcad00a1300 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fcad0074550 con 0x7fcad00a1300 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcac4001940 con 0x7fcad01341b0 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.829 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): dump: 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcac00018a0 con 0x7fcad00a1300 2024-09-15T20:50:41.830 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): dump: 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3347831638 0 0) 0x7fcad006ef60 con 0x7fcad013a790 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.831 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient(hunting): my global_id is 164203 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcabc004ec0 con 0x7fcad013a790 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4057576278 0 0) 0x7fcad0074550 con 0x7fcad00a1300 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 >> v1:172.21.15.146:6789/0 conn(0x7fcad013a790 legacy=0x7fcad01d0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 >> v1:172.21.15.78:6789/0 conn(0x7fcad01341b0 legacy=0x7fcad006e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 monclient: found mon.c 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcad0139f90 con 0x7fcad00a1300 2024-09-15T20:50:41.832 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 5 monclient: authenticate success, global_id 154307 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient: _renew_subs 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fcad0139740 con 0x7fcad00a1300 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 client.154307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient: _renew_subs 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7fcad0138ef0 con 0x7fcad00a1300 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 librados: init done 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fcac00047f0 con 0x7fcad00a1300 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.833 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fcac0004b40 con 0x7fcad00a1300 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcacd7fa640 4 set_mon_vals no callback set 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: dump: 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.834 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 client.154307.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 15 client.154307.objecter _session_command_op_assign -1 1 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 20 client.154307.objecter _get_session osd=-1 returning homeless 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 client.154307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7fcab0002920 con 0x7fcad00a1300 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fcac0010bb0 con 0x7fcad00a1300 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fcac004b4b0 con 0x7fcad00a1300 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.154307.objecter ms_dispatch 0x7fcad0000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 3 client.154307.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 3 client.154307.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.154307.objecter checking command 1 2024-09-15T20:50:41.835 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 20 client.154307.objecter _get_session s=0x7fcabc046010 osd=4 3 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 20 client.154307.objecter put_session s=0x7fcabc046010 osd=4 3 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 15 client.154307.objecter _session_command_op_remove -1 1 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 20 client.154307.objecter _get_session s=0x7fcabc046010 osd=4 3 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 20 client.154307.objecter get_session s=0x7fcabc046010 osd=4 3 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 15 client.154307.objecter _session_command_op_assign 4 1 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 20 client.154307.objecter put_session s=0x7fcabc046010 osd=4 4 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.154307.objecter _send_command 1 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.146:6808/2960966471 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcabc04a3d0 con 0x7fcabc0466c0 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 20 client.154307.objecter dump_active .. 0 homeless 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 3222990830 0 0) 0x7fcac004b900 con 0x7fcad00a1300 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcace7fc640 10 monclient: get_auth_request con 0x7fcabc03e180 auth_method 0 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcacf7fe640 10 monclient: get_auth_request con 0x7fcabc0466c0 auth_method 0 2024-09-15T20:50:41.836 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.154307.objecter ms_handle_connect 0x7fcabc03e180 2024-09-15T20:50:41.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.813+0000 7fcab77fe640 10 client.154307.objecter ms_handle_connect 0x7fcabc0466c0 2024-09-15T20:50:41.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.817+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== osd.4 v1:172.21.15.146:6808/2960966471 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7fcabc04a3d0 con 0x7fcabc0466c0 2024-09-15T20:50:41.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.817+0000 7fcab77fe640 10 client.154307.objecter ms_dispatch 0x7fcad0000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:50:41.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.817+0000 7fcab77fe640 10 client.154307.objecter _finish_command 1 = system:0 2024-09-15T20:50:41.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.817+0000 7fcab77fe640 20 client.154307.objecter put_session s=0x7fcabc046010 osd=4 4 2024-09-15T20:50:41.837 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.817+0000 7fcab77fe640 15 client.154307.objecter _session_command_op_remove 4 1 2024-09-15T20:50:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 10 client.154307.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T20:50:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 15 client.154307.objecter _session_command_op_assign -1 2 2024-09-15T20:50:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 20 client.154307.objecter _get_session s=0x7fcabc046010 osd=4 3 2024-09-15T20:50:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 20 client.154307.objecter put_session s=0x7fcabc046010 osd=4 3 2024-09-15T20:50:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 20 client.154307.objecter _get_session s=0x7fcabc046010 osd=4 3 2024-09-15T20:50:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 15 client.154307.objecter _session_command_op_remove -1 2 2024-09-15T20:50:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 20 client.154307.objecter get_session s=0x7fcabc046010 osd=4 3 2024-09-15T20:50:41.838 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 15 client.154307.objecter _session_command_op_assign 4 2 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 20 client.154307.objecter put_session s=0x7fcabc046010 osd=4 4 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 10 client.154307.objecter _send_command 2 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 --> v1:172.21.15.146:6808/2960966471 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fcab00012c0 con 0x7fcabc0466c0 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcab77fe640 1 -- 172.21.15.78:0/869983919 <== osd.4 v1:172.21.15.146:6808/2960966471 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7fcab00012c0 con 0x7fcabc0466c0 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcab77fe640 10 client.154307.objecter ms_dispatch 0x7fcad0000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcab77fe640 10 client.154307.objecter _finish_command 2 = system:0 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcab77fe640 20 client.154307.objecter put_session s=0x7fcabc046010 osd=4 4 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcab77fe640 15 client.154307.objecter _session_command_op_remove 4 2 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:50:41.839 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:41.840 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 10 librados: watch_flush enter 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 10 librados: watch_flush exit 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 10 client.154307.objecter close_session for osd.4 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 >> v1:172.21.15.146:6808/2960966471 conn(0x7fcabc0466c0 legacy=0x7fcabc048b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 20 client.154307.objecter put_session s=0x7fcabc046010 osd=4 1 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 20 client.154307.objecter shutdown clearing up homeless session... 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.829+0000 7fcad5779640 10 client.154307.objecter successfully canceled tick 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 >> v1:172.21.15.146:6812/435598642 conn(0x7fcabc03e180 legacy=0x7fcabc040640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 10 monclient: shutdown 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 >> v1:172.21.15.78:6790/0 conn(0x7fcad00a1300 legacy=0x7fcad006e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcacf7fe640 1 -- 172.21.15.78:0/869983919 reap_dead start 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 shutdown_connections 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 >> 172.21.15.78:0/869983919 conn(0x7fcad00a6240 msgr2=0x7fcad01337c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.841 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 shutdown_connections 2024-09-15T20:50:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 1 -- 172.21.15.78:0/869983919 wait complete. 2024-09-15T20:50:41.842 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.833+0000 7fcad5779640 1 librados: shutdown 2024-09-15T20:50:41.849 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient: get_monmap_and_config 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient: monmap: 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:41.957962+0000 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:41.957962+0000 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.956 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.957 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.957 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:41.957 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:41.957 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 1 Processor -- start 2024-09-15T20:50:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 1 -- start start 2024-09-15T20:50:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient: init 2024-09-15T20:50:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:50:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient(hunting): picked mon.noname-b con 0x7f5178135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient(hunting): picked mon.noname-a con 0x7f51780a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.964 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient(hunting): picked mon.noname-c con 0x7f517813c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51780745e0 con 0x7f51780a24e0 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f517806ecf0 con 0x7f517813c220 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f517806efd0 con 0x7f5178135b60 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient(hunting): authenticate will time out at 2302.842708s 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f5177fff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f51780a24e0 0x7f517813ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57984/0 (socket says 172.21.15.78:57984) 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f5177fff640 1 -- 172.21.15.78:0/3270761064 learned_addr learned my addr 172.21.15.78:0/3270761064 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3009940004 0 0) 0x7f51780745e0 con 0x7f51780a24e0 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.965 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient(hunting): my global_id is 144390 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5164003530 con 0x7f51780a24e0 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2206517152 0 0) 0x7f517806efd0 con 0x7f5178135b60 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient(hunting): my global_id is 154310 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f51780745e0 con 0x7f5178135b60 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 256738796 0 0) 0x7f51780745e0 con 0x7f5178135b60 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 >> v1:172.21.15.146:6789/0 conn(0x7f517813c220 legacy=0x7f517813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 >> v1:172.21.15.78:6789/0 conn(0x7f51780a24e0 legacy=0x7f517813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 monclient: found mon.noname-b 2024-09-15T20:50:41.966 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5178136650 con 0x7f5178135b60 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5160001680 con 0x7f5178135b60 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: dump: 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.967 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 5 monclient: authenticate success, global_id 154310 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5160001820 con 0x7f5178135b60 2024-09-15T20:50:41.968 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 1 -- 172.21.15.78:0/3270761064 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5160001c30 con 0x7f5178135b60 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 4 set_mon_vals no callback set 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f51777fe640 10 monclient: dump: 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.969 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 10 monclient: shutdown 2024-09-15T20:50:41.970 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.953+0000 7f517edbb640 1 -- 172.21.15.78:0/3270761064 >> v1:172.21.15.78:6790/0 conn(0x7f5178135b60 legacy=0x7f5178135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- 172.21.15.78:0/3270761064 shutdown_connections 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- 172.21.15.78:0/3270761064 >> 172.21.15.78:0/3270761064 conn(0x7f51780a63c0 msgr2=0x7f51780a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- 172.21.15.78:0/3270761064 shutdown_connections 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- 172.21.15.78:0/3270761064 wait complete. 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: build_initial_monmap 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: monmap: 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:41.971 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 librados: starting msgr at 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 librados: starting objecter 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 Processor -- start 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- start start 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 librados: setting wanted keys 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 librados: calling monclient init 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: init 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:41.972 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient(hunting): picked mon.noname-b con 0x7f51780a24e0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient(hunting): picked mon.noname-c con 0x7f5178135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient(hunting): picked mon.noname-a con 0x7f517813c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f517809d180 con 0x7f517813c220 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5178137f90 con 0x7f51780a24e0 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f51780728e0 con 0x7f5178135b60 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient(hunting): authenticate will time out at 2302.845453s 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517d358640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f517813c220 0x7f51781d2c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57988/0 (socket says 172.21.15.78:57988) 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517d358640 1 -- 172.21.15.78:0/65791327 learned_addr learned my addr 172.21.15.78:0/65791327 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 client.?.objecter ms_handle_connect 0x7f517813c220 2024-09-15T20:50:41.973 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 client.?.objecter ms_handle_connect 0x7f5178135b60 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2957810905 0 0) 0x7f517806ecf0 con 0x7f517813c220 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): my global_id is 144393 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5158004ec0 con 0x7f517813c220 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3614063137 0 0) 0x7f51780728e0 con 0x7f5178135b60 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): my global_id is 154313 2024-09-15T20:50:41.974 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f517806ecf0 con 0x7f5178135b60 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 client.?.objecter ms_handle_connect 0x7f51780a24e0 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2421923676 0 0) 0x7f5158004ec0 con 0x7f517813c220 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f51780728e0 con 0x7f517813c220 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f51700019c0 con 0x7f517813c220 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): dump: 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.975 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3192061047 0 0) 0x7f517806ecf0 con 0x7f5178135b60 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f5158004ec0 con 0x7f5178135b60 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5168001660 con 0x7f5178135b60 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): dump: 2024-09-15T20:50:41.976 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2621116158 0 0) 0x7f5178137f90 con 0x7f51780a24e0 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _negotiate 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient(hunting): my global_id is 164206 2024-09-15T20:50:41.977 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f517806ecf0 con 0x7f51780a24e0 2024-09-15T20:50:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 374860934 0 0) 0x7f51780728e0 con 0x7f517813c220 2024-09-15T20:50:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 >> v1:172.21.15.78:6790/0 conn(0x7f5178135b60 legacy=0x7f51780721d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.978 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 >> v1:172.21.15.146:6789/0 conn(0x7f51780a24e0 legacy=0x7f5178071ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient: _finish_hunting 0 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 monclient: found mon.a 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5178136d50 con 0x7f517813c220 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient: _finish_auth 0 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient: _check_auth_tickets 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 5 monclient: authenticate success, global_id 144393 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: _renew_subs 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f517813fa00 con 0x7f517813c220 2024-09-15T20:50:41.979 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 client.144393.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: _renew_subs 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f517813b490 con 0x7f517813c220 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 librados: init done 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5170001cc0 con 0x7f517813c220 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5170003a70 con 0x7f517813c220 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f5176ffd640 4 set_mon_vals no callback set 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 monclient: dump: 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.980 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f515c005090 con 0x7f517813c220 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f5170003c10 con 0x7f517813c220 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f517004b5c0 con 0x7f517813c220 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 10 client.144393.objecter ms_dispatch 0x7f5178000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 3 client.144393.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 3 client.144393.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:41.981 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f51757fa640 20 client.144393.objecter dump_active .. 0 homeless 2024-09-15T20:50:41.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.957+0000 7f517cb57640 10 monclient: get_auth_request con 0x7f515803e1f0 auth_method 0 2024-09-15T20:50:41.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.961+0000 7f51757fa640 10 client.144393.objecter ms_handle_connect 0x7f515803e1f0 2024-09-15T20:50:41.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.961+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f517809d180 con 0x7f517813c220 2024-09-15T20:50:41.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.961+0000 7f51757fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:41.982 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:41.961+0000 7f51757fa640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:42.131 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f515c000ba0 con 0x7f515803e1f0 2024-09-15T20:50:42.132 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f51757fa640 1 -- 172.21.15.78:0/65791327 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (unknown 3782719169 0 0) 0x7f515c000ba0 con 0x7f515803e1f0 2024-09-15T20:50:42.132 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 4 to deep-scrub 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 10 librados: watch_flush enter 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 10 librados: watch_flush exit 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 20 client.144393.objecter shutdown clearing up homeless session... 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 10 client.144393.objecter successfully canceled tick 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 >> v1:172.21.15.146:6812/435598642 conn(0x7f515803e1f0 legacy=0x7f51580406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 10 monclient: shutdown 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.129+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 >> v1:172.21.15.78:6789/0 conn(0x7f517813c220 legacy=0x7f51781d2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.135 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.133+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 shutdown_connections 2024-09-15T20:50:42.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.133+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 >> 172.21.15.78:0/65791327 conn(0x7f51780a63c0 msgr2=0x7f517812eb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:42.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.133+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 shutdown_connections 2024-09-15T20:50:42.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.133+0000 7f517edbb640 1 -- 172.21.15.78:0/65791327 wait complete. 2024-09-15T20:50:42.136 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.133+0000 7f517edbb640 1 librados: shutdown 2024-09-15T20:50:42.150 INFO:tasks.ceph:Scrubbing osd.5 2024-09-15T20:50:42.150 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-09-15T20:50:42.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient: get_monmap_and_config 2024-09-15T20:50:42.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient: build_initial_monmap 2024-09-15T20:50:42.260 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient: monmap: 2024-09-15T20:50:42.260 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:42.262431+0000 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:42.262431+0000 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:42.261 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 1 Processor -- start 2024-09-15T20:50:42.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 1 -- start start 2024-09-15T20:50:42.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient: init 2024-09-15T20:50:42.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:42.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:50:42.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient(hunting): picked mon.noname-b con 0x7f0be8137fc0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:42.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient(hunting): picked mon.noname-a con 0x7f0be80a0af0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:42.269 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient(hunting): picked mon.noname-c con 0x7f0be80a4d70 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be80a6290 con 0x7f0be80a0af0 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be80a1fa0 con 0x7f0be80a4d70 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be81383a0 con 0x7f0be8137fc0 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient(hunting): authenticate will time out at 2303.146932s 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bed643640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0be80a0af0 0x7f0be80a8bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:57994/0 (socket says 172.21.15.78:57994) 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bed643640 1 -- 172.21.15.78:0/2685522100 learned_addr learned my addr 172.21.15.78:0/2685522100 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bede44640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f0be8137fc0 0x7f0be80a84b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38582/0 (socket says 172.21.15.78:38582) 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 782392281 0 0) 0x7f0be80a6290 con 0x7f0be80a0af0 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.270 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient(hunting): my global_id is 144396 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bd4003530 con 0x7f0be80a0af0 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 477491057 0 0) 0x7f0be81383a0 con 0x7f0be8137fc0 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient(hunting): my global_id is 154316 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0be80a6290 con 0x7f0be8137fc0 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3275865740 0 0) 0x7f0be80a6290 con 0x7f0be8137fc0 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 >> v1:172.21.15.146:6789/0 conn(0x7f0be80a4d70 legacy=0x7f0be80a92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 >> v1:172.21.15.78:6789/0 conn(0x7f0be80a0af0 legacy=0x7f0be80a8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: _finish_hunting 0 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 monclient: found mon.noname-b 2024-09-15T20:50:42.271 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be8138570 con 0x7f0be8137fc0 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: _finish_auth 0 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: _check_auth_tickets 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 5 monclient: authenticate success, global_id 154316 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0be4001680 con 0x7f0be8137fc0 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: dump: 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.272 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0be40018a0 con 0x7f0be8137fc0 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 1 -- 172.21.15.78:0/2685522100 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0be4001cb0 con 0x7f0be8137fc0 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:42.273 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bf00a8640 4 set_mon_vals no callback set 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.257+0000 7f0bece42640 10 monclient: dump: 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.274 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: shutdown 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- 172.21.15.78:0/2685522100 >> v1:172.21.15.78:6790/0 conn(0x7f0be8137fc0 legacy=0x7f0be80a84b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- 172.21.15.78:0/2685522100 shutdown_connections 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- 172.21.15.78:0/2685522100 >> 172.21.15.78:0/2685522100 conn(0x7f0be80aa420 msgr2=0x7f0be80aa830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- 172.21.15.78:0/2685522100 shutdown_connections 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- 172.21.15.78:0/2685522100 wait complete. 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: build_initial_monmap 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: monmap: 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:42.275 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 librados: starting msgr at 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 librados: starting objecter 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 Processor -- start 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- start start 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 librados: setting wanted keys 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 librados: calling monclient init 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: init 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:42.276 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient(hunting): picked mon.noname-b con 0x7f0be80a0af0 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient(hunting): picked mon.noname-a con 0x7f0be80a4d70 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient(hunting): picked mon.noname-c con 0x7f0be8137fc0 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be8135930 con 0x7f0be80a4d70 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be8135c10 con 0x7f0be80a0af0 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0be8133070 con 0x7f0be8137fc0 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient(hunting): authenticate will time out at 2303.149960s 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bed643640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f0be80a4d70 0x7f0be8132960 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58008/0 (socket says 172.21.15.78:58008) 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bed643640 1 -- 172.21.15.78:0/640599547 learned_addr learned my addr 172.21.15.78:0/640599547 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 client.?.objecter ms_handle_connect 0x7f0be80a4d70 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 client.?.objecter ms_handle_connect 0x7f0be8137fc0 2024-09-15T20:50:42.277 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1828241630 0 0) 0x7f0be8135930 con 0x7f0be80a4d70 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): my global_id is 144399 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bcc004ec0 con 0x7f0be80a4d70 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2117101886 0 0) 0x7f0be8133070 con 0x7f0be8137fc0 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): my global_id is 154319 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0be8135930 con 0x7f0be8137fc0 2024-09-15T20:50:42.278 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 client.?.objecter ms_handle_connect 0x7f0be80a0af0 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 325375899 0 0) 0x7f0be8135930 con 0x7f0be8137fc0 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0be8133070 con 0x7f0be8137fc0 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0be0001c00 con 0x7f0be8137fc0 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): dump: 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.279 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3288307726 0 0) 0x7f0bcc004ec0 con 0x7f0be80a4d70 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f0be8135930 con 0x7f0be80a4d70 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0bdc0013a0 con 0x7f0be80a4d70 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): dump: 2024-09-15T20:50:42.280 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 357939665 0 0) 0x7f0be8135c10 con 0x7f0be80a0af0 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.281 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient(hunting): my global_id is 164212 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0bcc004ec0 con 0x7f0be80a0af0 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3583916862 0 0) 0x7f0be8133070 con 0x7f0be8137fc0 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 >> v1:172.21.15.146:6789/0 conn(0x7f0be80a0af0 legacy=0x7f0be8132250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 >> v1:172.21.15.78:6789/0 conn(0x7f0be80a4d70 legacy=0x7f0be8132960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 monclient: found mon.c 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0be8134910 con 0x7f0be8137fc0 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient: _finish_auth 0 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:42.282 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 5 monclient: authenticate success, global_id 154319 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: _renew_subs 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f0be8133bb0 con 0x7f0be8137fc0 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f0be0003980 con 0x7f0be8137fc0 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 client.154319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: _renew_subs 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f0be0003d90 con 0x7f0be8137fc0 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f0be8136310 con 0x7f0be8137fc0 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bdbfff640 4 set_mon_vals no callback set 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 librados: init done 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.283 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 10 monclient: dump: 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 client.154319.objecter _submit_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 15 client.154319.objecter _session_command_op_assign -1 1 2024-09-15T20:50:42.284 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 20 client.154319.objecter _get_session osd=-1 returning homeless 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 client.154319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.78:6790/0 -- mon_get_version(what=osdmap handle=1) -- 0x7f0be80a36c0 con 0x7f0be8137fc0 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.261+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f0be0003fb0 con 0x7f0be8137fc0 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f0be004a6b0 con 0x7f0be8137fc0 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 10 client.154319.objecter ms_dispatch 0x7f0be8000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 3 client.154319.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 3 client.154319.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 10 client.154319.objecter checking command 1 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 20 client.154319.objecter _get_session s=0x7f0bcc0460c0 osd=5 3 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 20 client.154319.objecter put_session s=0x7f0bcc0460c0 osd=5 3 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 15 client.154319.objecter _session_command_op_remove -1 1 2024-09-15T20:50:42.285 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 20 client.154319.objecter _get_session s=0x7f0bcc0460c0 osd=5 3 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 20 client.154319.objecter get_session s=0x7f0bcc0460c0 osd=5 3 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 15 client.154319.objecter _session_command_op_assign 5 1 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 20 client.154319.objecter put_session s=0x7f0bcc0460c0 osd=5 4 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 10 client.154319.objecter _send_command 1 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.146:6804/516207226 -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0bcc04a480 con 0x7f0bcc046770 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 20 client.154319.objecter dump_active .. 0 homeless 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_get_version_reply(handle=1 version=703) ==== 24+0+0 (unknown 3222990830 0 0) 0x7f0be004aa40 con 0x7f0be8137fc0 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 10 monclient: handle_get_version_reply finishing 1 version 703 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bede44640 10 monclient: get_auth_request con 0x7f0bcc03e1f0 auth_method 0 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bed643640 10 monclient: get_auth_request con 0x7f0bcc046770 auth_method 0 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 10 client.154319.objecter ms_handle_connect 0x7f0bcc046770 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 10 client.154319.objecter ms_handle_connect 0x7f0bcc03e1f0 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== osd.5 v1:172.21.15.146:6804/516207226 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (unknown 0 0 1302949928) 0x7f0bcc04a480 con 0x7f0bcc046770 2024-09-15T20:50:42.286 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 10 client.154319.objecter ms_dispatch 0x7f0be8000ba0 command_reply(tid 1: 0 ) 2024-09-15T20:50:42.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 10 client.154319.objecter _finish_command 1 = system:0 2024-09-15T20:50:42.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 20 client.154319.objecter put_session s=0x7f0bcc0460c0 osd=5 4 2024-09-15T20:50:42.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.265+0000 7f0bda7fc640 15 client.154319.objecter _session_command_op_remove 5 1 2024-09-15T20:50:42.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 10 client.154319.objecter _submit_command 2 [{"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}] 2024-09-15T20:50:42.287 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 15 client.154319.objecter _session_command_op_assign -1 2 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 20 client.154319.objecter _get_session s=0x7f0bcc0460c0 osd=5 3 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 20 client.154319.objecter put_session s=0x7f0bcc0460c0 osd=5 3 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 20 client.154319.objecter _get_session s=0x7f0bcc0460c0 osd=5 3 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 15 client.154319.objecter _session_command_op_remove -1 2 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 20 client.154319.objecter get_session s=0x7f0bcc0460c0 osd=5 3 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 15 client.154319.objecter _session_command_op_assign 5 2 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 20 client.154319.objecter put_session s=0x7f0bcc0460c0 osd=5 4 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 10 client.154319.objecter _send_command 2 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 --> v1:172.21.15.146:6804/516207226 -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f0be8136010 con 0x7f0bcc046770 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bda7fc640 1 -- 172.21.15.78:0/640599547 <== osd.5 v1:172.21.15.146:6804/516207226 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (unknown 0 0 119112422) 0x7f0bdc007820 con 0x7f0bcc046770 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bda7fc640 10 client.154319.objecter ms_dispatch 0x7f0be8000ba0 command_reply(tid 2: 0 ) 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bda7fc640 10 client.154319.objecter _finish_command 2 = system:0 2024-09-15T20:50:42.288 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bda7fc640 20 client.154319.objecter put_session s=0x7f0bcc0460c0 osd=5 4 2024-09-15T20:50:42.289 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bda7fc640 15 client.154319.objecter _session_command_op_remove 5 2 2024-09-15T20:50:42.289 INFO:teuthology.orchestra.run.smithi078.stdout:{ 2024-09-15T20:50:42.289 INFO:teuthology.orchestra.run.smithi078.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-15T20:50:42.289 INFO:teuthology.orchestra.run.smithi078.stdout:} 2024-09-15T20:50:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 10 librados: watch_flush enter 2024-09-15T20:50:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 10 librados: watch_flush exit 2024-09-15T20:50:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 10 client.154319.objecter close_session for osd.5 2024-09-15T20:50:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 >> v1:172.21.15.146:6804/516207226 conn(0x7f0bcc046770 legacy=0x7f0bcc048c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.290 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 20 client.154319.objecter put_session s=0x7f0bcc0460c0 osd=5 1 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 20 client.154319.objecter shutdown clearing up homeless session... 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 10 client.154319.objecter successfully canceled tick 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 >> v1:172.21.15.146:6812/435598642 conn(0x7f0bcc03e1f0 legacy=0x7f0bcc0406b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 10 monclient: shutdown 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 >> v1:172.21.15.78:6790/0 conn(0x7f0be8137fc0 legacy=0x7f0be81d4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bee645640 1 -- 172.21.15.78:0/640599547 reap_dead start 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 shutdown_connections 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.277+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 >> 172.21.15.78:0/640599547 conn(0x7f0be80aa420 msgr2=0x7f0be80a2300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.281+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 shutdown_connections 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.281+0000 7f0bf00a8640 1 -- 172.21.15.78:0/640599547 wait complete. 2024-09-15T20:50:42.291 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.281+0000 7f0bf00a8640 1 librados: shutdown 2024-09-15T20:50:42.296 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-09-15T20:50:42.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.393+0000 7f53189bd640 10 monclient: get_monmap_and_config 2024-09-15T20:50:42.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.393+0000 7f53189bd640 10 monclient: build_initial_monmap 2024-09-15T20:50:42.398 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.393+0000 7f53189bd640 10 monclient: monmap: 2024-09-15T20:50:42.398 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:42.398 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.399 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:42.400550+0000 2024-09-15T20:50:42.399 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:42.400550+0000 2024-09-15T20:50:42.399 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:42.399 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.399 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:42.399 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:42.399 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:42.399 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 Processor -- start 2024-09-15T20:50:42.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- start start 2024-09-15T20:50:42.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: init 2024-09-15T20:50:42.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:42.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: _add_conns ranks=[1,0,2] 2024-09-15T20:50:42.405 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): picked mon.noname-c con 0x7f5314135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): picked mon.noname-a con 0x7f53140a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): picked mon.noname-b con 0x7f531413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53140745e0 con 0x7f53140a24e0 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f531406ecf0 con 0x7f5314135b60 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f531406efd0 con 0x7f531413c220 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): authenticate will time out at 2303.284818s 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530bfff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f53140a24e0 0x7f531413ab00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58024/0 (socket says 172.21.15.78:58024) 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530bfff640 1 -- 172.21.15.78:0/4017307086 learned_addr learned my addr 172.21.15.78:0/4017307086 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952337879 0 0) 0x7f53140745e0 con 0x7f53140a24e0 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.406 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient(hunting): my global_id is 144402 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52f4003530 con 0x7f53140a24e0 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2584337424 0 0) 0x7f531406efd0 con 0x7f531413c220 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient(hunting): my global_id is 154322 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53140745e0 con 0x7f531413c220 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2653499041 0 0) 0x7f53140745e0 con 0x7f531413c220 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 >> v1:172.21.15.146:6789/0 conn(0x7f5314135b60 legacy=0x7f5314135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.407 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 >> v1:172.21.15.78:6789/0 conn(0x7f53140a24e0 legacy=0x7f531413ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: _finish_hunting 0 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 monclient: found mon.noname-b 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5314136650 con 0x7f531413c220 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: _finish_auth 0 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: _check_auth_tickets 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f530c001930 con 0x7f531413c220 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 5 monclient: authenticate success, global_id 154322 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.408 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: dump: 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:42.409 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f530c001b50 con 0x7f531413c220 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 1 -- 172.21.15.78:0/4017307086 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f530c001f60 con 0x7f531413c220 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 4 set_mon_vals no callback set 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f5311d9b640 10 monclient: dump: 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.410 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: shutdown 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- 172.21.15.78:0/4017307086 >> v1:172.21.15.78:6790/0 conn(0x7f531413c220 legacy=0x7f531413e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- 172.21.15.78:0/4017307086 shutdown_connections 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- 172.21.15.78:0/4017307086 >> 172.21.15.78:0/4017307086 conn(0x7f53140a63c0 msgr2=0x7f53140a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- 172.21.15.78:0/4017307086 shutdown_connections 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- 172.21.15.78:0/4017307086 wait complete. 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: build_initial_monmap 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: monmap: 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:42.411 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 librados: starting msgr at 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 librados: starting objecter 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 Processor -- start 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- start start 2024-09-15T20:50:42.412 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 librados: setting wanted keys 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 librados: calling monclient init 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: init 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient: _add_conns ranks=[0,1,2] 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): picked mon.noname-a con 0x7f53140a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): picked mon.noname-b con 0x7f5314135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): picked mon.noname-c con 0x7f531413c220 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53141385e0 con 0x7f53140a24e0 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f531409d180 con 0x7f5314135b60 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f53140737d0 con 0x7f531413c220 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f53189bd640 10 monclient(hunting): authenticate will time out at 2303.287667s 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f531259c640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f53140a24e0 0x7f5314071d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58026/0 (socket says 172.21.15.78:58026) 2024-09-15T20:50:42.413 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f531259c640 1 -- 172.21.15.78:0/2208853324 learned_addr learned my addr 172.21.15.78:0/2208853324 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 client.?.objecter ms_handle_connect 0x7f53140a24e0 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 client.?.objecter ms_handle_connect 0x7f531413c220 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3836739934 0 0) 0x7f53141385e0 con 0x7f53140a24e0 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 monclient(hunting): my global_id is 144405 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f52f8004ec0 con 0x7f53140a24e0 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3841188255 0 0) 0x7f53140737d0 con 0x7f531413c220 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.414 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 10 monclient(hunting): my global_id is 154325 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.397+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f53141385e0 con 0x7f531413c220 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 2610114226 0 0) 0x7f52f8004ec0 con 0x7f53140a24e0 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f53140737d0 con 0x7f53140a24e0 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5304001a40 con 0x7f53140a24e0 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient(hunting): dump: 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.415 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 client.?.objecter ms_handle_connect 0x7f5314135b60 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3007395070 0 0) 0x7f53141385e0 con 0x7f531413c220 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f52f8004ec0 con 0x7f531413c220 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f530c001ea0 con 0x7f531413c220 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:42.416 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient(hunting): dump: 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 3641202415 0 0) 0x7f53140737d0 con 0x7f53140a24e0 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 >> v1:172.21.15.78:6790/0 conn(0x7f531413c220 legacy=0x7f5314072bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 >> v1:172.21.15.146:6789/0 conn(0x7f5314135b60 legacy=0x7f53140724a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.417 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient: _finish_hunting 0 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 monclient: found mon.a 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5314137d40 con 0x7f53140a24e0 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient: _finish_auth 0 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient: _check_auth_tickets 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 5 monclient: authenticate success, global_id 144405 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 10 monclient: _renew_subs 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f531413b8b0 con 0x7f53140a24e0 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 10 client.144405.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:42.418 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 10 monclient: _renew_subs 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f53141d3df0 con 0x7f53140a24e0 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 1 librados: init done 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f5304001d40 con 0x7f53140a24e0 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f5304003a70 con 0x7f53140a24e0 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f531159a640 4 set_mon_vals no callback set 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 monclient: dump: 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.419 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52ec005090 con 0x7f53140a24e0 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f5304003c10 con 0x7f53140a24e0 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f530404b6d0 con 0x7f53140a24e0 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 client.144405.objecter ms_dispatch 0x7f5314000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:42.420 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 3 client.144405.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:42.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 3 client.144405.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:42.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 20 client.144405.objecter dump_active .. 0 homeless 2024-09-15T20:50:42.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530bfff640 10 monclient: get_auth_request con 0x7f52f803e080 auth_method 0 2024-09-15T20:50:42.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.401+0000 7f530affd640 10 client.144405.objecter ms_handle_connect 0x7f52f803e080 2024-09-15T20:50:42.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.405+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f5304016c50 con 0x7f53140a24e0 2024-09-15T20:50:42.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.405+0000 7f530affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:42.421 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.405+0000 7f530affd640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:42.570 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.565+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f52ec000ba0 con 0x7f52f803e080 2024-09-15T20:50:42.571 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f530affd640 1 -- 172.21.15.78:0/2208853324 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (unknown 2264484800 0 0) 0x7f52ec000ba0 con 0x7f52f803e080 2024-09-15T20:50:42.571 INFO:teuthology.orchestra.run.smithi078.stderr:instructed osd(s) 5 to deep-scrub 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 10 librados: watch_flush enter 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 10 librados: watch_flush exit 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 20 client.144405.objecter shutdown clearing up homeless session... 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 10 client.144405.objecter successfully canceled tick 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 >> v1:172.21.15.146:6812/435598642 conn(0x7f52f803e080 legacy=0x7f52f8040540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 10 monclient: shutdown 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 >> v1:172.21.15.78:6789/0 conn(0x7f53140a24e0 legacy=0x7f5314071d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 shutdown_connections 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 >> 172.21.15.78:0/2208853324 conn(0x7f53140a63c0 msgr2=0x7f53140a3d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 shutdown_connections 2024-09-15T20:50:42.574 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 1 -- 172.21.15.78:0/2208853324 wait complete. 2024-09-15T20:50:42.575 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.569+0000 7f53189bd640 1 librados: shutdown 2024-09-15T20:50:42.588 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:50:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.697+0000 7fb686793640 10 monclient: get_monmap_and_config 2024-09-15T20:50:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.697+0000 7fb686793640 10 monclient: build_initial_monmap 2024-09-15T20:50:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.697+0000 7fb686793640 10 monclient: monmap: 2024-09-15T20:50:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:50:42.703821+0000 2024-09-15T20:50:42.702 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:50:42.703821+0000 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.697+0000 7fb686793640 1 Processor -- start 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.697+0000 7fb686793640 1 -- start start 2024-09-15T20:50:42.703 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.697+0000 7fb686793640 10 monclient: init 2024-09-15T20:50:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: _add_conns ranks=[2,0,1] 2024-09-15T20:50:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): picked mon.noname-b con 0x7fb680135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): picked mon.noname-a con 0x7fb6800a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): picked mon.noname-c con 0x7fb68013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6800745e0 con 0x7fb6800a24e0 2024-09-15T20:50:42.709 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68006ecf0 con 0x7fb68013c220 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68006efd0 con 0x7fb680135b60 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): authenticate will time out at 2303.588479s 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67ffff640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7fb680135b60 0x7fb680135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38626/0 (socket says 172.21.15.78:38626) 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67ffff640 1 -- 172.21.15.78:0/4191885597 learned_addr learned my addr 172.21.15.78:0/4191885597 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 143154764 0 0) 0x7fb68006efd0 con 0x7fb680135b60 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient(hunting): my global_id is 154328 2024-09-15T20:50:42.710 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb66c003530 con 0x7fb680135b60 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 308616309 0 0) 0x7fb6800745e0 con 0x7fb6800a24e0 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient(hunting): my global_id is 144408 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb68006efd0 con 0x7fb6800a24e0 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4141605180 0 0) 0x7fb68006efd0 con 0x7fb6800a24e0 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 >> v1:172.21.15.78:6790/0 conn(0x7fb680135b60 legacy=0x7fb680135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 >> v1:172.21.15.146:6789/0 conn(0x7fb68013c220 legacy=0x7fb68013e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: _finish_hunting 0 2024-09-15T20:50:42.711 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 monclient: found mon.noname-a 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb680136650 con 0x7fb6800a24e0 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: _finish_auth 0 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: _check_auth_tickets 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 5 monclient: authenticate success, global_id 144408 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb678001a00 con 0x7fb6800a24e0 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: dump: 2024-09-15T20:50:42.712 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb678001ba0 con 0x7fb6800a24e0 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: get_monmap_and_config success 2024-09-15T20:50:42.713 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 1 -- 172.21.15.78:0/4191885597 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb678001f30 con 0x7fb6800a24e0 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 4 set_mon_vals no callback set 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67effd640 10 monclient: dump: 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.714 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: shutdown 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- 172.21.15.78:0/4191885597 >> v1:172.21.15.78:6789/0 conn(0x7fb6800a24e0 legacy=0x7fb68013ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- 172.21.15.78:0/4191885597 shutdown_connections 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- 172.21.15.78:0/4191885597 >> 172.21.15.78:0/4191885597 conn(0x7fb6800a63c0 msgr2=0x7fb6800a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- 172.21.15.78:0/4191885597 shutdown_connections 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- 172.21.15.78:0/4191885597 wait complete. 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: build_initial_monmap 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: monmap: 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:50:42.715 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 librados: starting msgr at 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 librados: starting objecter 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 Processor -- start 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- start start 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 librados: setting wanted keys 2024-09-15T20:50:42.716 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 librados: calling monclient init 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: init 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: _reopen_session rank -1 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): picked mon.noname-a con 0x7fb6800a24e0 addr v1:172.21.15.78:6789/0 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): picked mon.noname-c con 0x7fb680135b60 addr v1:172.21.15.78:6790/0 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): picked mon.noname-b con 0x7fb68013c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb680073150 con 0x7fb6800a24e0 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb6800a3840 con 0x7fb68013c220 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68009d180 con 0x7fb680135b60 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): _renew_subs 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb686793640 10 monclient(hunting): authenticate will time out at 2303.591277s 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67ffff640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7fb6800a24e0 0x7fb680072330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:58040/0 (socket says 172.21.15.78:58040) 2024-09-15T20:50:42.717 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67ffff640 1 -- 172.21.15.78:0/1796728618 learned_addr learned my addr 172.21.15.78:0/1796728618 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67f7fe640 1 --1- 172.21.15.78:0/1796728618 >> v1:172.21.15.78:6790/0 conn(0x7fb680135b60 0x7fb680072a40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:38640/0 (socket says 172.21.15.78:38640) 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 client.?.objecter ms_handle_connect 0x7fb6800a24e0 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 client.?.objecter ms_handle_connect 0x7fb680135b60 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 952310057 0 0) 0x7fb68009d180 con 0x7fb680135b60 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): my global_id is 154331 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb660004ef0 con 0x7fb680135b60 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3733255980 0 0) 0x7fb680073150 con 0x7fb6800a24e0 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): _negotiate 2024-09-15T20:50:42.718 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): my global_id is 144411 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb68009d180 con 0x7fb6800a24e0 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 client.?.objecter ms_handle_connect 0x7fb68013c220 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 client.?.objecter resend_mon_ops 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4121649827 0 0) 0x7fb660004ef0 con 0x7fb680135b60 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb680073150 con 0x7fb680135b60 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3741227620 0 0) 0x7fb68009d180 con 0x7fb6800a24e0 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7fb660004ef0 con 0x7fb6800a24e0 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb678001980 con 0x7fb680135b60 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): dump: 2024-09-15T20:50:42.719 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb6680013a0 con 0x7fb6800a24e0 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:42.720 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.701+0000 7fb67cff9640 10 monclient(hunting): dump: 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 4063066300 0 0) 0x7fb660004ef0 con 0x7fb6800a24e0 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 >> v1:172.21.15.78:6790/0 conn(0x7fb680135b60 legacy=0x7fb680072a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 >> v1:172.21.15.146:6789/0 conn(0x7fb68013c220 legacy=0x7fb6801d4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.721 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 monclient: _finish_hunting 0 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 monclient: found mon.a 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb68011e880 con 0x7fb6800a24e0 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 monclient: _finish_auth 0 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 monclient: _check_auth_tickets 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 5 monclient: authenticate success, global_id 144411 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 10 monclient: _renew_subs 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.722 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7fb68011eb60 con 0x7fb6800a24e0 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 10 client.144411.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 10 monclient: _renew_subs 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7fb68011d5c0 con 0x7fb6800a24e0 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 1 librados: init done 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7fb6680047f0 con 0x7fb6800a24e0 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 monclient: handle_config config(0 keys) 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7fb668004b40 con 0x7fb6800a24e0 2024-09-15T20:50:42.723 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67e7fc640 4 set_mon_vals no callback set 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 monclient: dump: 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:50:42.724 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb664005090 con 0x7fb6800a24e0 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7fb668010bb0 con 0x7fb6800a24e0 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7fb66804b5d0 con 0x7fb6800a24e0 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 client.144411.objecter ms_dispatch 0x7fb680000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 3 client.144411.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 3 client.144411.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 20 client.144411.objecter dump_active .. 0 homeless 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67f7fe640 10 monclient: get_auth_request con 0x7fb66003de80 auth_method 0 2024-09-15T20:50:42.725 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.705+0000 7fb67cff9640 10 client.144411.objecter ms_handle_connect 0x7fb66003de80 2024-09-15T20:50:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.709+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7fb668016ab0 con 0x7fb6800a24e0 2024-09-15T20:50:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.709+0000 7fb67cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:50:42.726 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.709+0000 7fb67cff9640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:50:42.876 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.873+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb664000ba0 con 0x7fb66003de80 2024-09-15T20:50:42.880 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.877+0000 7fb67cff9640 1 -- 172.21.15.78:0/1796728618 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114126 (unknown 2965378022 0 1613821485) 0x7fb664000ba0 con 0x7fb66003de80 2024-09-15T20:50:42.881 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:50:42.882 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:50:42.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 10 librados: watch_flush enter 2024-09-15T20:50:42.883 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 10 librados: watch_flush exit 2024-09-15T20:50:42.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 20 client.144411.objecter shutdown clearing up homeless session... 2024-09-15T20:50:42.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 10 client.144411.objecter successfully canceled tick 2024-09-15T20:50:42.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 >> v1:172.21.15.146:6812/435598642 conn(0x7fb66003de80 legacy=0x7fb660040340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 10 monclient: shutdown 2024-09-15T20:50:42.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:50:42.884 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 >> v1:172.21.15.78:6789/0 conn(0x7fb6800a24e0 legacy=0x7fb680072330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:50:42.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 shutdown_connections 2024-09-15T20:50:42.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 >> 172.21.15.78:0/1796728618 conn(0x7fb6800a63c0 msgr2=0x7fb68013f400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:50:42.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 shutdown_connections 2024-09-15T20:50:42.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 1 -- 172.21.15.78:0/1796728618 wait complete. 2024-09-15T20:50:42.885 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:50:42.881+0000 7fb686793640 1 librados: shutdown 2024-09-15T20:50:42.900 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":1270,"stamp":"2024-09-15T20:50:41.049315+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":409016,"kb_used_data":31672,"kb_used_omap":9,"kb_used_meta":377206,"kb_avail":565822024,"statfs":{"total":579820584960,"available":579401752576,"internally_reserved":0,"allocated":32432128,"data_stored":14246060,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":386259616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004022"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529766+0000","last_change":"2024-09-15T20:32:49.923652+0000","last_active":"2024-09-15T20:49:19.529766+0000","last_peered":"2024-09-15T20:49:19.529766+0000","last_clean":"2024-09-15T20:49:19.529766+0000","last_became_active":"2024-09-15T20:32:49.923378+0000","last_became_peered":"2024-09-15T20:32:49.923378+0000","last_unstale":"2024-09-15T20:49:19.529766+0000","last_undegraded":"2024-09-15T20:49:19.529766+0000","last_fullsized":"2024-09-15T20:49:19.529766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:22:44.936546+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365025+0000","last_change":"2024-09-15T20:32:49.869206+0000","last_active":"2024-09-15T20:49:20.365025+0000","last_peered":"2024-09-15T20:49:20.365025+0000","last_clean":"2024-09-15T20:49:20.365025+0000","last_became_active":"2024-09-15T20:32:49.868460+0000","last_became_peered":"2024-09-15T20:32:49.868460+0000","last_unstale":"2024-09-15T20:49:20.365025+0000","last_undegraded":"2024-09-15T20:49:20.365025+0000","last_fullsized":"2024-09-15T20:49:20.365025+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:52.936657+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949298+0000","last_change":"2024-09-15T20:32:49.923995+0000","last_active":"2024-09-15T20:49:19.949298+0000","last_peered":"2024-09-15T20:49:19.949298+0000","last_clean":"2024-09-15T20:49:19.949298+0000","last_became_active":"2024-09-15T20:32:49.923706+0000","last_became_peered":"2024-09-15T20:32:49.923706+0000","last_unstale":"2024-09-15T20:49:19.949298+0000","last_undegraded":"2024-09-15T20:49:19.949298+0000","last_fullsized":"2024-09-15T20:49:19.949298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:11:23.148031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173672+0000","last_change":"2024-09-15T20:32:49.831383+0000","last_active":"2024-09-15T20:49:18.173672+0000","last_peered":"2024-09-15T20:49:18.173672+0000","last_clean":"2024-09-15T20:49:18.173672+0000","last_became_active":"2024-09-15T20:32:49.830739+0000","last_became_peered":"2024-09-15T20:32:49.830739+0000","last_unstale":"2024-09-15T20:49:18.173672+0000","last_undegraded":"2024-09-15T20:49:18.173672+0000","last_fullsized":"2024-09-15T20:49:18.173672+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:11:38.383945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949357+0000","last_change":"2024-09-15T20:32:49.923640+0000","last_active":"2024-09-15T20:49:19.949357+0000","last_peered":"2024-09-15T20:49:19.949357+0000","last_clean":"2024-09-15T20:49:19.949357+0000","last_became_active":"2024-09-15T20:32:49.921234+0000","last_became_peered":"2024-09-15T20:32:49.921234+0000","last_unstale":"2024-09-15T20:49:19.949357+0000","last_undegraded":"2024-09-15T20:49:19.949357+0000","last_fullsized":"2024-09-15T20:49:19.949357+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:03:20.711668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173971+0000","last_change":"2024-09-15T20:32:49.866040+0000","last_active":"2024-09-15T20:49:18.173971+0000","last_peered":"2024-09-15T20:49:18.173971+0000","last_clean":"2024-09-15T20:49:18.173971+0000","last_became_active":"2024-09-15T20:32:49.864741+0000","last_became_peered":"2024-09-15T20:32:49.864741+0000","last_unstale":"2024-09-15T20:49:18.173971+0000","last_undegraded":"2024-09-15T20:49:18.173971+0000","last_fullsized":"2024-09-15T20:49:18.173971+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:31:00.623365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365414+0000","last_change":"2024-09-15T20:32:49.889906+0000","last_active":"2024-09-15T20:49:20.365414+0000","last_peered":"2024-09-15T20:49:20.365414+0000","last_clean":"2024-09-15T20:49:20.365414+0000","last_became_active":"2024-09-15T20:32:49.887399+0000","last_became_peered":"2024-09-15T20:32:49.887399+0000","last_unstale":"2024-09-15T20:49:20.365414+0000","last_undegraded":"2024-09-15T20:49:20.365414+0000","last_fullsized":"2024-09-15T20:49:20.365414+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:02:18.324444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365329+0000","last_change":"2024-09-15T20:32:49.929668+0000","last_active":"2024-09-15T20:49:20.365329+0000","last_peered":"2024-09-15T20:49:20.365329+0000","last_clean":"2024-09-15T20:49:20.365329+0000","last_became_active":"2024-09-15T20:32:49.929513+0000","last_became_peered":"2024-09-15T20:32:49.929513+0000","last_unstale":"2024-09-15T20:49:20.365329+0000","last_undegraded":"2024-09-15T20:49:20.365329+0000","last_fullsized":"2024-09-15T20:49:20.365329+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:53:07.079742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"22'7","reported_seq":1354,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949429+0000","last_change":"2024-09-15T20:32:49.912474+0000","last_active":"2024-09-15T20:49:19.949429+0000","last_peered":"2024-09-15T20:49:19.949429+0000","last_clean":"2024-09-15T20:49:19.949429+0000","last_became_active":"2024-09-15T20:32:49.912168+0000","last_became_peered":"2024-09-15T20:32:49.912168+0000","last_unstale":"2024-09-15T20:49:19.949429+0000","last_undegraded":"2024-09-15T20:49:19.949429+0000","last_fullsized":"2024-09-15T20:49:19.949429+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:42:18.704520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173397+0000","last_change":"2024-09-15T20:32:49.905970+0000","last_active":"2024-09-15T20:49:18.173397+0000","last_peered":"2024-09-15T20:49:18.173397+0000","last_clean":"2024-09-15T20:49:18.173397+0000","last_became_active":"2024-09-15T20:32:49.903676+0000","last_became_peered":"2024-09-15T20:32:49.903676+0000","last_unstale":"2024-09-15T20:49:18.173397+0000","last_undegraded":"2024-09-15T20:49:18.173397+0000","last_fullsized":"2024-09-15T20:49:18.173397+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:44:35.092934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317692+0000","last_change":"2024-09-15T20:32:49.832927+0000","last_active":"2024-09-15T20:49:19.317692+0000","last_peered":"2024-09-15T20:49:19.317692+0000","last_clean":"2024-09-15T20:49:19.317692+0000","last_became_active":"2024-09-15T20:32:49.832239+0000","last_became_peered":"2024-09-15T20:32:49.832239+0000","last_unstale":"2024-09-15T20:49:19.317692+0000","last_undegraded":"2024-09-15T20:49:19.317692+0000","last_fullsized":"2024-09-15T20:49:19.317692+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:34:24.870248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173579+0000","last_change":"2024-09-15T20:32:49.963674+0000","last_active":"2024-09-15T20:49:18.173579+0000","last_peered":"2024-09-15T20:49:18.173579+0000","last_clean":"2024-09-15T20:49:18.173579+0000","last_became_active":"2024-09-15T20:32:49.963404+0000","last_became_peered":"2024-09-15T20:32:49.963404+0000","last_unstale":"2024-09-15T20:49:18.173579+0000","last_undegraded":"2024-09-15T20:49:18.173579+0000","last_fullsized":"2024-09-15T20:49:18.173579+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:18.112614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"15'72","reported_seq":1442,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317359+0000","last_change":"2024-09-15T20:31:40.658325+0000","last_active":"2024-09-15T20:49:19.317359+0000","last_peered":"2024-09-15T20:49:19.317359+0000","last_clean":"2024-09-15T20:49:19.317359+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:49:19.317359+0000","last_undegraded":"2024-09-15T20:49:19.317359+0000","last_fullsized":"2024-09-15T20:49:19.317359+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:38.726961+0000","last_clean_scrub_stamp":"2024-09-15T20:31:38.726961+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:23:41.743437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"17'1","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365458+0000","last_change":"2024-09-15T20:31:47.833645+0000","last_active":"2024-09-15T20:49:20.365458+0000","last_peered":"2024-09-15T20:49:20.365458+0000","last_clean":"2024-09-15T20:49:20.365458+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:49:20.365458+0000","last_undegraded":"2024-09-15T20:49:20.365458+0000","last_fullsized":"2024-09-15T20:49:20.365458+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:34:24.343248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1344,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529672+0000","last_change":"2024-09-15T20:32:49.945387+0000","last_active":"2024-09-15T20:49:19.529672+0000","last_peered":"2024-09-15T20:49:19.529672+0000","last_clean":"2024-09-15T20:49:19.529672+0000","last_became_active":"2024-09-15T20:32:49.943293+0000","last_became_peered":"2024-09-15T20:32:49.943293+0000","last_unstale":"2024-09-15T20:49:19.529672+0000","last_undegraded":"2024-09-15T20:49:19.529672+0000","last_fullsized":"2024-09-15T20:49:19.529672+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:38:58.278471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317229+0000","last_change":"2024-09-15T20:31:47.825565+0000","last_active":"2024-09-15T20:49:19.317229+0000","last_peered":"2024-09-15T20:49:19.317229+0000","last_clean":"2024-09-15T20:49:19.317229+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:49:19.317229+0000","last_undegraded":"2024-09-15T20:49:19.317229+0000","last_fullsized":"2024-09-15T20:49:19.317229+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:49:12.374739+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949050+0000","last_change":"2024-09-15T20:32:49.929346+0000","last_active":"2024-09-15T20:49:19.949050+0000","last_peered":"2024-09-15T20:49:19.949050+0000","last_clean":"2024-09-15T20:49:19.949050+0000","last_became_active":"2024-09-15T20:32:49.926970+0000","last_became_peered":"2024-09-15T20:32:49.926970+0000","last_unstale":"2024-09-15T20:49:19.949050+0000","last_undegraded":"2024-09-15T20:49:19.949050+0000","last_fullsized":"2024-09-15T20:49:19.949050+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:57:20.804839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1360,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529794+0000","last_change":"2024-09-15T20:31:47.969706+0000","last_active":"2024-09-15T20:49:19.529794+0000","last_peered":"2024-09-15T20:49:19.529794+0000","last_clean":"2024-09-15T20:49:19.529794+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:49:19.529794+0000","last_undegraded":"2024-09-15T20:49:19.529794+0000","last_fullsized":"2024-09-15T20:49:19.529794+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:16:48.645009+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317504+0000","last_change":"2024-09-15T20:32:49.858231+0000","last_active":"2024-09-15T20:49:19.317504+0000","last_peered":"2024-09-15T20:49:19.317504+0000","last_clean":"2024-09-15T20:49:19.317504+0000","last_became_active":"2024-09-15T20:32:49.857975+0000","last_became_peered":"2024-09-15T20:32:49.857975+0000","last_unstale":"2024-09-15T20:49:19.317504+0000","last_undegraded":"2024-09-15T20:49:19.317504+0000","last_fullsized":"2024-09-15T20:49:19.317504+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:16:52.881614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"19'2","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365169+0000","last_change":"2024-09-15T20:31:47.834864+0000","last_active":"2024-09-15T20:49:20.365169+0000","last_peered":"2024-09-15T20:49:20.365169+0000","last_clean":"2024-09-15T20:49:20.365169+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:49:20.365169+0000","last_undegraded":"2024-09-15T20:49:20.365169+0000","last_fullsized":"2024-09-15T20:49:20.365169+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:24:56.096771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949156+0000","last_change":"2024-09-15T20:32:49.950036+0000","last_active":"2024-09-15T20:49:19.949156+0000","last_peered":"2024-09-15T20:49:19.949156+0000","last_clean":"2024-09-15T20:49:19.949156+0000","last_became_active":"2024-09-15T20:32:49.949390+0000","last_became_peered":"2024-09-15T20:32:49.949390+0000","last_unstale":"2024-09-15T20:49:19.949156+0000","last_undegraded":"2024-09-15T20:49:19.949156+0000","last_fullsized":"2024-09-15T20:49:19.949156+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:26:57.150999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.174094+0000","last_change":"2024-09-15T20:31:47.838237+0000","last_active":"2024-09-15T20:49:18.174094+0000","last_peered":"2024-09-15T20:49:18.174094+0000","last_clean":"2024-09-15T20:49:18.174094+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:49:18.174094+0000","last_undegraded":"2024-09-15T20:49:18.174094+0000","last_fullsized":"2024-09-15T20:49:18.174094+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:33:49.045779+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.365184+0000","last_change":"2024-09-15T20:32:49.909995+0000","last_active":"2024-09-15T20:49:20.365184+0000","last_peered":"2024-09-15T20:49:20.365184+0000","last_clean":"2024-09-15T20:49:20.365184+0000","last_became_active":"2024-09-15T20:32:49.909430+0000","last_became_peered":"2024-09-15T20:32:49.909430+0000","last_unstale":"2024-09-15T20:49:20.365184+0000","last_undegraded":"2024-09-15T20:49:20.365184+0000","last_fullsized":"2024-09-15T20:49:20.365184+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:59:09.239728+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317818+0000","last_change":"2024-09-15T20:31:47.825665+0000","last_active":"2024-09-15T20:49:19.317818+0000","last_peered":"2024-09-15T20:49:19.317818+0000","last_clean":"2024-09-15T20:49:19.317818+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:49:19.317818+0000","last_undegraded":"2024-09-15T20:49:19.317818+0000","last_fullsized":"2024-09-15T20:49:19.317818+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:40:22.110997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317704+0000","last_change":"2024-09-15T20:32:49.893856+0000","last_active":"2024-09-15T20:49:19.317704+0000","last_peered":"2024-09-15T20:49:19.317704+0000","last_clean":"2024-09-15T20:49:19.317704+0000","last_became_active":"2024-09-15T20:32:49.893603+0000","last_became_peered":"2024-09-15T20:32:49.893603+0000","last_unstale":"2024-09-15T20:49:19.317704+0000","last_undegraded":"2024-09-15T20:49:19.317704+0000","last_fullsized":"2024-09-15T20:49:19.317704+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:18:19.480960+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173996+0000","last_change":"2024-09-15T20:31:47.838456+0000","last_active":"2024-09-15T20:49:18.173996+0000","last_peered":"2024-09-15T20:49:18.173996+0000","last_clean":"2024-09-15T20:49:18.173996+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:49:18.173996+0000","last_undegraded":"2024-09-15T20:49:18.173996+0000","last_fullsized":"2024-09-15T20:49:18.173996+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:31:54.370290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317442+0000","last_change":"2024-09-15T20:32:49.908877+0000","last_active":"2024-09-15T20:49:19.317442+0000","last_peered":"2024-09-15T20:49:19.317442+0000","last_clean":"2024-09-15T20:49:19.317442+0000","last_became_active":"2024-09-15T20:32:49.908352+0000","last_became_peered":"2024-09-15T20:32:49.908352+0000","last_unstale":"2024-09-15T20:49:19.317442+0000","last_undegraded":"2024-09-15T20:49:19.317442+0000","last_fullsized":"2024-09-15T20:49:19.317442+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:01:49.714804+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317897+0000","last_change":"2024-09-15T20:31:47.826026+0000","last_active":"2024-09-15T20:49:19.317897+0000","last_peered":"2024-09-15T20:49:19.317897+0000","last_clean":"2024-09-15T20:49:19.317897+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:49:19.317897+0000","last_undegraded":"2024-09-15T20:49:19.317897+0000","last_fullsized":"2024-09-15T20:49:19.317897+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:31:44.773733+0000","last_clean_scrub_stamp":"2024-09-15T20:31:44.773733+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:12:19.315897+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.529924+0000","last_change":"2024-09-15T20:32:49.973245+0000","last_active":"2024-09-15T20:49:19.529924+0000","last_peered":"2024-09-15T20:49:19.529924+0000","last_clean":"2024-09-15T20:49:19.529924+0000","last_became_active":"2024-09-15T20:32:49.969795+0000","last_became_peered":"2024-09-15T20:32:49.969795+0000","last_unstale":"2024-09-15T20:49:19.529924+0000","last_undegraded":"2024-09-15T20:49:19.529924+0000","last_fullsized":"2024-09-15T20:49:19.529924+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:23:10.478700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949261+0000","last_change":"2024-09-15T20:32:49.948981+0000","last_active":"2024-09-15T20:49:19.949261+0000","last_peered":"2024-09-15T20:49:19.949261+0000","last_clean":"2024-09-15T20:49:19.949261+0000","last_became_active":"2024-09-15T20:32:49.948678+0000","last_became_peered":"2024-09-15T20:32:49.948678+0000","last_unstale":"2024-09-15T20:49:19.949261+0000","last_undegraded":"2024-09-15T20:49:19.949261+0000","last_fullsized":"2024-09-15T20:49:19.949261+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:33:32.657477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.174262+0000","last_change":"2024-09-15T20:32:49.875011+0000","last_active":"2024-09-15T20:49:18.174262+0000","last_peered":"2024-09-15T20:49:18.174262+0000","last_clean":"2024-09-15T20:49:18.174262+0000","last_became_active":"2024-09-15T20:32:49.872991+0000","last_became_peered":"2024-09-15T20:32:49.872991+0000","last_unstale":"2024-09-15T20:49:18.174262+0000","last_undegraded":"2024-09-15T20:49:18.174262+0000","last_fullsized":"2024-09-15T20:49:18.174262+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:51:49.787166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949272+0000","last_change":"2024-09-15T20:32:50.022200+0000","last_active":"2024-09-15T20:49:19.949272+0000","last_peered":"2024-09-15T20:49:19.949272+0000","last_clean":"2024-09-15T20:49:19.949272+0000","last_became_active":"2024-09-15T20:32:50.021808+0000","last_became_peered":"2024-09-15T20:32:50.021808+0000","last_unstale":"2024-09-15T20:49:19.949272+0000","last_undegraded":"2024-09-15T20:49:19.949272+0000","last_fullsized":"2024-09-15T20:49:19.949272+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:28:02.730241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.949602+0000","last_change":"2024-09-15T20:32:49.946363+0000","last_active":"2024-09-15T20:49:19.949602+0000","last_peered":"2024-09-15T20:49:19.949602+0000","last_clean":"2024-09-15T20:49:19.949602+0000","last_became_active":"2024-09-15T20:32:49.937453+0000","last_became_peered":"2024-09-15T20:32:49.937453+0000","last_unstale":"2024-09-15T20:49:19.949602+0000","last_undegraded":"2024-09-15T20:49:19.949602+0000","last_fullsized":"2024-09-15T20:49:19.949602+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:12.866160+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1332,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.364943+0000","last_change":"2024-09-15T20:32:49.906716+0000","last_active":"2024-09-15T20:49:20.364943+0000","last_peered":"2024-09-15T20:49:20.364943+0000","last_clean":"2024-09-15T20:49:20.364943+0000","last_became_active":"2024-09-15T20:32:49.906485+0000","last_became_peered":"2024-09-15T20:32:49.906485+0000","last_unstale":"2024-09-15T20:49:20.364943+0000","last_undegraded":"2024-09-15T20:49:20.364943+0000","last_fullsized":"2024-09-15T20:49:20.364943+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:03:16.163113+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317437+0000","last_change":"2024-09-15T20:32:49.846500+0000","last_active":"2024-09-15T20:49:19.317437+0000","last_peered":"2024-09-15T20:49:19.317437+0000","last_clean":"2024-09-15T20:49:19.317437+0000","last_became_active":"2024-09-15T20:32:49.844315+0000","last_became_peered":"2024-09-15T20:32:49.844315+0000","last_unstale":"2024-09-15T20:49:19.317437+0000","last_undegraded":"2024-09-15T20:49:19.317437+0000","last_fullsized":"2024-09-15T20:49:19.317437+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:43:04.920505+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327733+0000","last_change":"2024-09-15T20:32:49.935233+0000","last_active":"2024-09-15T20:49:20.327733+0000","last_peered":"2024-09-15T20:49:20.327733+0000","last_clean":"2024-09-15T20:49:20.327733+0000","last_became_active":"2024-09-15T20:32:49.930427+0000","last_became_peered":"2024-09-15T20:32:49.930427+0000","last_unstale":"2024-09-15T20:49:20.327733+0000","last_undegraded":"2024-09-15T20:49:20.327733+0000","last_fullsized":"2024-09-15T20:49:20.327733+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:00:31.898076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317298+0000","last_change":"2024-09-15T20:32:49.880288+0000","last_active":"2024-09-15T20:49:19.317298+0000","last_peered":"2024-09-15T20:49:19.317298+0000","last_clean":"2024-09-15T20:49:19.317298+0000","last_became_active":"2024-09-15T20:32:49.880025+0000","last_became_peered":"2024-09-15T20:32:49.880025+0000","last_unstale":"2024-09-15T20:49:19.317298+0000","last_undegraded":"2024-09-15T20:49:19.317298+0000","last_fullsized":"2024-09-15T20:49:19.317298+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T02:10:36.755630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1327,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327939+0000","last_change":"2024-09-15T20:32:49.942013+0000","last_active":"2024-09-15T20:49:20.327939+0000","last_peered":"2024-09-15T20:49:20.327939+0000","last_clean":"2024-09-15T20:49:20.327939+0000","last_became_active":"2024-09-15T20:32:49.939238+0000","last_became_peered":"2024-09-15T20:32:49.939238+0000","last_unstale":"2024-09-15T20:49:20.327939+0000","last_undegraded":"2024-09-15T20:49:20.327939+0000","last_fullsized":"2024-09-15T20:49:20.327939+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:23:42.080704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:19.317993+0000","last_change":"2024-09-15T20:32:49.906455+0000","last_active":"2024-09-15T20:49:19.317993+0000","last_peered":"2024-09-15T20:49:19.317993+0000","last_clean":"2024-09-15T20:49:19.317993+0000","last_became_active":"2024-09-15T20:32:49.903971+0000","last_became_peered":"2024-09-15T20:32:49.903971+0000","last_unstale":"2024-09-15T20:49:19.317993+0000","last_undegraded":"2024-09-15T20:49:19.317993+0000","last_fullsized":"2024-09-15T20:49:19.317993+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:03:18.879453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1333,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:18.173724+0000","last_change":"2024-09-15T20:32:49.910696+0000","last_active":"2024-09-15T20:49:18.173724+0000","last_peered":"2024-09-15T20:49:18.173724+0000","last_clean":"2024-09-15T20:49:18.173724+0000","last_became_active":"2024-09-15T20:32:49.909018+0000","last_became_peered":"2024-09-15T20:32:49.909018+0000","last_unstale":"2024-09-15T20:49:18.173724+0000","last_undegraded":"2024-09-15T20:49:18.173724+0000","last_fullsized":"2024-09-15T20:49:18.173724+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:33:03.321005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1326,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:49:20.327891+0000","last_change":"2024-09-15T20:32:49.943386+0000","last_active":"2024-09-15T20:49:20.327891+0000","last_peered":"2024-09-15T20:49:20.327891+0000","last_clean":"2024-09-15T20:49:20.327891+0000","last_became_active":"2024-09-15T20:32:49.942857+0000","last_became_peered":"2024-09-15T20:32:49.942857+0000","last_unstale":"2024-09-15T20:49:20.327891+0000","last_undegraded":"2024-09-15T20:49:20.327891+0000","last_fullsized":"2024-09-15T20:49:20.327891+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:32:48.730090+0000","last_clean_scrub_stamp":"2024-09-15T20:32:48.730090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T05:13:01.229283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607783,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72548,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299292,"statfs":{"total":96636764160,"available":96562475008,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607782,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5256,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5382144,"data_stored":2198026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607783,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50336,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94321504,"statfs":{"total":96636764160,"available":96585220096,"internally_reserved":0,"allocated":5386240,"data_stored":2198031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607782,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72604,"kb_used_data":5320,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299236,"statfs":{"total":96636764160,"available":96562417664,"internally_reserved":0,"allocated":5447680,"data_stored":2726959,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640487,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68440,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303400,"statfs":{"total":96636764160,"available":96566681600,"internally_reserved":0,"allocated":5443584,"data_stored":2726954,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640487,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72548,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299292,"statfs":{"total":96636764160,"available":96562475008,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:50:42.904 INFO:tasks.ceph:pgid 9.19 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.904 INFO:tasks.ceph:pgid 9.18 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.904 INFO:tasks.ceph:pgid 9.1b last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.1a last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.5 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.4 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.7 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.6 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.1 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.0 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.3 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.905 INFO:tasks.ceph:pgid 9.2 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-15T20:31:38.726961+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-15T20:31:44.773733+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 9.8 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-15T20:31:44.773733+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 9.b last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-15T20:31:44.773733+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 9.a last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-15T20:31:44.773733+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.906 INFO:tasks.ceph:pgid 9.9 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-15T20:31:44.773733+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 9.f last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-15T20:31:44.773733+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 9.e last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-15T20:31:44.773733+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 9.d last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-15T20:31:44.773733+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=31, tm_sec=44, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 9.c last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.907 INFO:tasks.ceph:pgid 9.1e last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.1f last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.1c last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.1d last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.12 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.13 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.10 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.11 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.16 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.908 INFO:tasks.ceph:pgid 9.17 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.909 INFO:tasks.ceph:pgid 9.14 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.909 INFO:tasks.ceph:pgid 9.15 last_scrub_stamp 2024-09-15T20:32:48.730090+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=32, tm_sec=48, tm_wday=6, tm_yday=259, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=15, tm_hour=20, tm_min=50, tm_sec=38, tm_wday=6, tm_yday=259, tm_isdst=0) 2024-09-15T20:50:42.909 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-15T20:51:02.910 DEBUG:teuthology.orchestra.run.smithi078:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-09-15T20:51:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.040+0000 7f7f501f9640 10 monclient: get_monmap_and_config 2024-09-15T20:51:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.040+0000 7f7f501f9640 10 monclient: build_initial_monmap 2024-09-15T20:51:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.040+0000 7f7f501f9640 10 monclient: monmap: 2024-09-15T20:51:03.042 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:51:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:51:03.043912+0000 2024-09-15T20:51:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:51:03.043912+0000 2024-09-15T20:51:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:51:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.043 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:51:03.044 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:51:03.044 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:51:03.044 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 Processor -- start 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- start start 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient: init 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient: _reopen_session rank -1 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient: _add_conns ranks=[0,2,1] 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient(hunting): picked mon.noname-a con 0x7f7f48135b60 addr v1:172.21.15.78:6789/0 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient(hunting): picked mon.noname-b con 0x7f7f480a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient(hunting): picked mon.noname-c con 0x7f7f4813c220 addr v1:172.21.15.146:6789/0 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f480745e0 con 0x7f7f48135b60 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4806ecf0 con 0x7f7f4813c220 2024-09-15T20:51:03.058 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4806efd0 con 0x7f7f480a24e0 2024-09-15T20:51:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient(hunting): _renew_subs 2024-09-15T20:51:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient(hunting): authenticate will time out at 2323.929044s 2024-09-15T20:51:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4df95640 1 --1- >> v1:172.21.15.78:6789/0 conn(0x7f7f48135b60 0x7f7f48135f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45062/0 (socket says 172.21.15.78:45062) 2024-09-15T20:51:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4df95640 1 -- 172.21.15.78:0/943514274 learned_addr learned my addr 172.21.15.78:0/943514274 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:51:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 506691110 0 0) 0x7f7f480745e0 con 0x7f7f48135b60 2024-09-15T20:51:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.059 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): my global_id is 144414 2024-09-15T20:51:03.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f34003530 con 0x7f7f48135b60 2024-09-15T20:51:03.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 197673765 0 0) 0x7f7f4806efd0 con 0x7f7f480a24e0 2024-09-15T20:51:03.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.060 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): my global_id is 154334 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f480745e0 con 0x7f7f480a24e0 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3016939559 0 0) 0x7f7f4806ecf0 con 0x7f7f4813c220 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient(hunting): my global_id is 164227 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f4806efd0 con 0x7f7f4813c220 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2092858540 0 0) 0x7f7f34003530 con 0x7f7f48135b60 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 >> v1:172.21.15.78:6790/0 conn(0x7f7f480a24e0 legacy=0x7f7f4813ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.061 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 >> v1:172.21.15.146:6789/0 conn(0x7f7f4813c220 legacy=0x7f7f4813e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: _finish_hunting 0 2024-09-15T20:51:03.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 monclient: found mon.noname-a 2024-09-15T20:51:03.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:51:03.062 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.78:6789/0 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f48136650 con 0x7f7f48135b60 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: _finish_auth 0 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: _check_auth_tickets 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7f38001620 con 0x7f7f48135b60 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 5 monclient: authenticate success, global_id 144414 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: dump: 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.063 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.064 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 <== mon.0 v1:172.21.15.78:6789/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f380017c0 con 0x7f7f48135b60 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: handle_config config(0 keys) 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient: get_monmap_and_config success 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 4 set_mon_vals no callback set 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 1 -- 172.21.15.78:0/943514274 <== mon.0 v1:172.21.15.78:6789/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7f38001c50 con 0x7f7f48135b60 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f4cf93640 10 monclient: dump: 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.065 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.066 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.066 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.066 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.066 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.066 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.066 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.067 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.067 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient: shutdown 2024-09-15T20:51:03.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:51:03.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- 172.21.15.78:0/943514274 >> v1:172.21.15.78:6789/0 conn(0x7f7f48135b60 legacy=0x7f7f48135f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- 172.21.15.78:0/943514274 shutdown_connections 2024-09-15T20:51:03.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- 172.21.15.78:0/943514274 >> 172.21.15.78:0/943514274 conn(0x7f7f480a63c0 msgr2=0x7f7f480a67d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:51:03.067 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- 172.21.15.78:0/943514274 shutdown_connections 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- 172.21.15.78:0/943514274 wait complete. 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient: build_initial_monmap 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient: monmap: 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:51:03.068 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 librados: starting msgr at 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 librados: starting objecter 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 Processor -- start 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 -- start start 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 librados: setting wanted keys 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 1 librados: calling monclient init 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.044+0000 7f7f501f9640 10 monclient: init 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient: _reopen_session rank -1 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient(hunting): picked mon.noname-c con 0x7f7f480a24e0 addr v1:172.21.15.78:6790/0 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient(hunting): picked mon.noname-b con 0x7f7f48135b60 addr v1:172.21.15.146:6789/0 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient(hunting): picked mon.noname-a con 0x7f7f4813c220 addr v1:172.21.15.78:6789/0 2024-09-15T20:51:03.069 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f480734f0 con 0x7f7f4813c220 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f48136d30 con 0x7f7f48135b60 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f4809d180 con 0x7f7f480a24e0 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient(hunting): _renew_subs 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient(hunting): authenticate will time out at 2323.932259s 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f4df95640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f7f480a24e0 0x7f7f481cae20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56650/0 (socket says 172.21.15.78:56650) 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f4df95640 1 -- 172.21.15.78:0/3103821769 learned_addr learned my addr 172.21.15.78:0/3103821769 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 client.?.objecter ms_handle_connect 0x7f7f480a24e0 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:51:03.070 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 client.?.objecter ms_handle_connect 0x7f7f4813c220 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2438274492 0 0) 0x7f7f480734f0 con 0x7f7f4813c220 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): my global_id is 144417 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f2c004ef0 con 0x7f7f4813c220 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3308243501 0 0) 0x7f7f4809d180 con 0x7f7f480a24e0 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.071 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): my global_id is 154337 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f480734f0 con 0x7f7f480a24e0 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 client.?.objecter ms_handle_connect 0x7f7f48135b60 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 client.?.objecter resend_mon_ops 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3703142072 0 0) 0x7f7f2c004ef0 con 0x7f7f4813c220 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f4809d180 con 0x7f7f4813c220 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7f40001c80 con 0x7f7f4813c220 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): dump: 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.072 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3372937565 0 0) 0x7f7f480734f0 con 0x7f7f480a24e0 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f7f2c004ef0 con 0x7f7f480a24e0 2024-09-15T20:51:03.073 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7f38001c50 con 0x7f7f480a24e0 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): dump: 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.074 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.075 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.075 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.075 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.075 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2310465145 0 0) 0x7f7f48136d30 con 0x7f7f48135b60 2024-09-15T20:51:03.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.075 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient(hunting): my global_id is 164230 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f480734f0 con 0x7f7f48135b60 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1948273957 0 0) 0x7f7f4809d180 con 0x7f7f4813c220 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 >> v1:172.21.15.78:6790/0 conn(0x7f7f480a24e0 legacy=0x7f7f481cae20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 >> v1:172.21.15.146:6789/0 conn(0x7f7f48135b60 legacy=0x7f7f481cfe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient: _finish_hunting 0 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 monclient: found mon.a 2024-09-15T20:51:03.076 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.78:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f48137c90 con 0x7f7f4813c220 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient: _finish_auth 0 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient: _check_auth_tickets 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 5 monclient: authenticate success, global_id 144417 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient: _renew_subs 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.78:6789/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f7f48072a40 con 0x7f7f4813c220 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 client.144417.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient: _renew_subs 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:51:03.077 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.78:6789/0 -- mon_subscribe({osdmap=0}) -- 0x7f7f48138b20 con 0x7f7f4813c220 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 1 librados: init done 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f7f400039a0 con 0x7f7f4813c220 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f7f40003cf0 con 0x7f7f4813c220 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3ffff640 4 set_mon_vals no callback set 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 monclient: dump: 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.078 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.78:6789/0 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.78:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f30005090 con 0x7f7f4813c220 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f7f4000fd60 con 0x7f7f4813c220 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f7f4004a570 con 0x7f7f4813c220 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 client.144417.objecter ms_dispatch 0x7f7f48000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 3 client.144417.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 3 client.144417.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:51:03.079 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 20 client.144417.objecter dump_active .. 0 homeless 2024-09-15T20:51:03.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f4df95640 10 monclient: get_auth_request con 0x7f7f2c03e180 auth_method 0 2024-09-15T20:51:03.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.048+0000 7f7f3e7fc640 10 client.144417.objecter ms_handle_connect 0x7f7f2c03e180 2024-09-15T20:51:03.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.052+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mon.0 v1:172.21.15.78:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f7f40015ad0 con 0x7f7f4813c220 2024-09-15T20:51:03.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.052+0000 7f7f3e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:51:03.080 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.052+0000 7f7f3e7fc640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:51:03.216 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.216+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 --> v1:172.21.15.146:6812/435598642 -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f7f30000ba0 con 0x7f7f2c03e180 2024-09-15T20:51:03.222 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.220+0000 7f7f3e7fc640 1 -- 172.21.15.78:0/3103821769 <== mgr.4104 v1:172.21.15.146:6812/435598642 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+114148 (unknown 2965378022 0 1140852447) 0x7f7f30000ba0 con 0x7f7f2c03e180 2024-09-15T20:51:03.223 INFO:teuthology.orchestra.run.smithi078.stdout: 2024-09-15T20:51:03.225 INFO:teuthology.orchestra.run.smithi078.stderr:dumped all 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 10 librados: watch_flush enter 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 10 librados: watch_flush exit 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 20 client.144417.objecter shutdown clearing up homeless session... 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 10 client.144417.objecter successfully canceled tick 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 >> v1:172.21.15.146:6812/435598642 conn(0x7f7f2c03e180 legacy=0x7f7f2c040640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 10 monclient: shutdown 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 >> v1:172.21.15.78:6789/0 conn(0x7f7f4813c220 legacy=0x7f7f481d3530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 shutdown_connections 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 >> 172.21.15.78:0/3103821769 conn(0x7f7f480a63c0 msgr2=0x7f7f4812ebc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 shutdown_connections 2024-09-15T20:51:03.226 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 1 -- 172.21.15.78:0/3103821769 wait complete. 2024-09-15T20:51:03.227 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.224+0000 7f7f501f9640 1 librados: shutdown 2024-09-15T20:51:03.240 INFO:teuthology.orchestra.run.smithi078.stdout:{"pg_ready":true,"pg_map":{"version":1281,"stamp":"2024-09-15T20:51:03.056297+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590392,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":72,"num_read_kb":61,"num_write":110,"num_write_kb":1637,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":87,"ondisk_log_size":87,"up":82,"acting":82,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":82,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":6,"kb":566231040,"kb_used":408984,"kb_used_data":31672,"kb_used_omap":9,"kb_used_meta":377206,"kb_avail":565822056,"statfs":{"total":579820584960,"available":579401785344,"internally_reserved":0,"allocated":32432128,"data_stored":14246060,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":9568,"internal_metadata":386259616},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003790"},"pg_stats":[{"pgid":"9.19","version":"0'0","reported_seq":1351,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:44.525226+0000","last_change":"2024-09-15T20:50:44.525226+0000","last_active":"2024-09-15T20:50:44.525226+0000","last_peered":"2024-09-15T20:50:44.525226+0000","last_clean":"2024-09-15T20:50:44.525226+0000","last_became_active":"2024-09-15T20:32:49.923378+0000","last_became_peered":"2024-09-15T20:32:49.923378+0000","last_unstale":"2024-09-15T20:50:44.525226+0000","last_undegraded":"2024-09-15T20:50:44.525226+0000","last_fullsized":"2024-09-15T20:50:44.525226+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:44.525069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:44.525069+0000","last_clean_scrub_stamp":"2024-09-15T20:50:44.525069+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:12:25.498775+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.18","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:49.252464+0000","last_change":"2024-09-15T20:50:49.252464+0000","last_active":"2024-09-15T20:50:49.252464+0000","last_peered":"2024-09-15T20:50:49.252464+0000","last_clean":"2024-09-15T20:50:49.252464+0000","last_became_active":"2024-09-15T20:32:49.868460+0000","last_became_peered":"2024-09-15T20:32:49.868460+0000","last_unstale":"2024-09-15T20:50:49.252464+0000","last_undegraded":"2024-09-15T20:50:49.252464+0000","last_fullsized":"2024-09-15T20:50:49.252464+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:49.252297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:49.252297+0000","last_clean_scrub_stamp":"2024-09-15T20:50:49.252297+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:27:41.733174+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1b","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:45.524160+0000","last_change":"2024-09-15T20:50:45.524160+0000","last_active":"2024-09-15T20:50:45.524160+0000","last_peered":"2024-09-15T20:50:45.524160+0000","last_clean":"2024-09-15T20:50:45.524160+0000","last_became_active":"2024-09-15T20:32:49.923706+0000","last_became_peered":"2024-09-15T20:32:49.923706+0000","last_unstale":"2024-09-15T20:50:45.524160+0000","last_undegraded":"2024-09-15T20:50:45.524160+0000","last_fullsized":"2024-09-15T20:50:45.524160+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:45.524090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:45.524090+0000","last_clean_scrub_stamp":"2024-09-15T20:50:45.524090+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:14:56.697383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1a","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:48.555247+0000","last_change":"2024-09-15T20:50:48.555247+0000","last_active":"2024-09-15T20:50:48.555247+0000","last_peered":"2024-09-15T20:50:48.555247+0000","last_clean":"2024-09-15T20:50:48.555247+0000","last_became_active":"2024-09-15T20:32:49.830739+0000","last_became_peered":"2024-09-15T20:32:49.830739+0000","last_unstale":"2024-09-15T20:50:48.555247+0000","last_undegraded":"2024-09-15T20:50:48.555247+0000","last_fullsized":"2024-09-15T20:50:48.555247+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:48.555079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:48.555079+0000","last_clean_scrub_stamp":"2024-09-15T20:50:48.555079+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:43:12.273204+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.5","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:40.527933+0000","last_change":"2024-09-15T20:50:40.527933+0000","last_active":"2024-09-15T20:50:40.527933+0000","last_peered":"2024-09-15T20:50:40.527933+0000","last_clean":"2024-09-15T20:50:40.527933+0000","last_became_active":"2024-09-15T20:32:49.921234+0000","last_became_peered":"2024-09-15T20:32:49.921234+0000","last_unstale":"2024-09-15T20:50:40.527933+0000","last_undegraded":"2024-09-15T20:50:40.527933+0000","last_fullsized":"2024-09-15T20:50:40.527933+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:40.527858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:40.527858+0000","last_clean_scrub_stamp":"2024-09-15T20:50:40.527858+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:47:11.869582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.4","version":"0'0","reported_seq":1342,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:45.614305+0000","last_change":"2024-09-15T20:50:45.614305+0000","last_active":"2024-09-15T20:50:45.614305+0000","last_peered":"2024-09-15T20:50:45.614305+0000","last_clean":"2024-09-15T20:50:45.614305+0000","last_became_active":"2024-09-15T20:32:49.864741+0000","last_became_peered":"2024-09-15T20:32:49.864741+0000","last_unstale":"2024-09-15T20:50:45.614305+0000","last_undegraded":"2024-09-15T20:50:45.614305+0000","last_fullsized":"2024-09-15T20:50:45.614305+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:45.614140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:45.614140+0000","last_clean_scrub_stamp":"2024-09-15T20:50:45.614140+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:41:23.776945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.7","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:45.268922+0000","last_change":"2024-09-15T20:50:45.268922+0000","last_active":"2024-09-15T20:50:45.268922+0000","last_peered":"2024-09-15T20:50:45.268922+0000","last_clean":"2024-09-15T20:50:45.268922+0000","last_became_active":"2024-09-15T20:32:49.887399+0000","last_became_peered":"2024-09-15T20:32:49.887399+0000","last_unstale":"2024-09-15T20:50:45.268922+0000","last_undegraded":"2024-09-15T20:50:45.268922+0000","last_fullsized":"2024-09-15T20:50:45.268922+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:45.268764+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:45.268764+0000","last_clean_scrub_stamp":"2024-09-15T20:50:45.268764+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:37:33.772715+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.6","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:44.229097+0000","last_change":"2024-09-15T20:50:44.229097+0000","last_active":"2024-09-15T20:50:44.229097+0000","last_peered":"2024-09-15T20:50:44.229097+0000","last_clean":"2024-09-15T20:50:44.229097+0000","last_became_active":"2024-09-15T20:32:49.929513+0000","last_became_peered":"2024-09-15T20:32:49.929513+0000","last_unstale":"2024-09-15T20:50:44.229097+0000","last_undegraded":"2024-09-15T20:50:44.229097+0000","last_fullsized":"2024-09-15T20:50:44.229097+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:44.228914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:44.228914+0000","last_clean_scrub_stamp":"2024-09-15T20:50:44.228914+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:07:06.611167+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.1","version":"703'8","reported_seq":1363,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:41.527462+0000","last_change":"2024-09-15T20:50:41.527462+0000","last_active":"2024-09-15T20:50:41.527462+0000","last_peered":"2024-09-15T20:50:41.527462+0000","last_clean":"2024-09-15T20:50:41.527462+0000","last_became_active":"2024-09-15T20:32:49.912168+0000","last_became_peered":"2024-09-15T20:32:49.912168+0000","last_unstale":"2024-09-15T20:50:41.527462+0000","last_undegraded":"2024-09-15T20:50:41.527462+0000","last_fullsized":"2024-09-15T20:50:41.527462+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'8","last_scrub_stamp":"2024-09-15T20:50:41.527399+0000","last_deep_scrub":"703'8","last_deep_scrub_stamp":"2024-09-15T20:50:41.527399+0000","last_clean_scrub_stamp":"2024-09-15T20:50:41.527399+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:42:21.958416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.0","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:42.497970+0000","last_change":"2024-09-15T20:50:42.497820+0000","last_active":"2024-09-15T20:50:42.497970+0000","last_peered":"2024-09-15T20:50:42.497970+0000","last_clean":"2024-09-15T20:50:42.497970+0000","last_became_active":"2024-09-15T20:32:49.903676+0000","last_became_peered":"2024-09-15T20:32:49.903676+0000","last_unstale":"2024-09-15T20:50:42.497970+0000","last_undegraded":"2024-09-15T20:50:42.497970+0000","last_fullsized":"2024-09-15T20:50:42.497970+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:42.497667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:42.497667+0000","last_clean_scrub_stamp":"2024-09-15T20:50:42.497667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:48:14.121137+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.3","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:43.205393+0000","last_change":"2024-09-15T20:50:43.205393+0000","last_active":"2024-09-15T20:50:43.205393+0000","last_peered":"2024-09-15T20:50:43.205393+0000","last_clean":"2024-09-15T20:50:43.205393+0000","last_became_active":"2024-09-15T20:32:49.832239+0000","last_became_peered":"2024-09-15T20:32:49.832239+0000","last_unstale":"2024-09-15T20:50:43.205393+0000","last_undegraded":"2024-09-15T20:50:43.205393+0000","last_fullsized":"2024-09-15T20:50:43.205393+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:43.205236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:43.205236+0000","last_clean_scrub_stamp":"2024-09-15T20:50:43.205236+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:46:36.434347+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.2","version":"0'0","reported_seq":1342,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:44.584344+0000","last_change":"2024-09-15T20:50:44.584344+0000","last_active":"2024-09-15T20:50:44.584344+0000","last_peered":"2024-09-15T20:50:44.584344+0000","last_clean":"2024-09-15T20:50:44.584344+0000","last_became_active":"2024-09-15T20:32:49.963404+0000","last_became_peered":"2024-09-15T20:32:49.963404+0000","last_unstale":"2024-09-15T20:50:44.584344+0000","last_undegraded":"2024-09-15T20:50:44.584344+0000","last_fullsized":"2024-09-15T20:50:44.584344+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:44.584170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:44.584170+0000","last_clean_scrub_stamp":"2024-09-15T20:50:44.584170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:42:09.593602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"703'74","reported_seq":1452,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:42.243603+0000","last_change":"2024-09-15T20:50:42.243603+0000","last_active":"2024-09-15T20:50:42.243603+0000","last_peered":"2024-09-15T20:50:42.243603+0000","last_clean":"2024-09-15T20:50:42.243603+0000","last_became_active":"2024-09-15T20:31:40.656536+0000","last_became_peered":"2024-09-15T20:31:40.656536+0000","last_unstale":"2024-09-15T20:50:42.243603+0000","last_undegraded":"2024-09-15T20:50:42.243603+0000","last_fullsized":"2024-09-15T20:50:42.243603+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":13,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'74","last_scrub_stamp":"2024-09-15T20:50:42.243436+0000","last_deep_scrub":"703'74","last_deep_scrub_stamp":"2024-09-15T20:50:42.243436+0000","last_clean_scrub_stamp":"2024-09-15T20:50:42.243436+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:34:20.792949+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"703'2","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:46.233215+0000","last_change":"2024-09-15T20:50:46.233215+0000","last_active":"2024-09-15T20:50:46.233215+0000","last_peered":"2024-09-15T20:50:46.233215+0000","last_clean":"2024-09-15T20:50:46.233215+0000","last_became_active":"2024-09-15T20:31:45.799052+0000","last_became_peered":"2024-09-15T20:31:45.799052+0000","last_unstale":"2024-09-15T20:50:46.233215+0000","last_undegraded":"2024-09-15T20:50:46.233215+0000","last_fullsized":"2024-09-15T20:50:46.233215+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'2","last_scrub_stamp":"2024-09-15T20:50:46.233053+0000","last_deep_scrub":"703'2","last_deep_scrub_stamp":"2024-09-15T20:50:46.233053+0000","last_clean_scrub_stamp":"2024-09-15T20:50:46.233053+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:21:47.264953+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0011824489999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.8","version":"0'0","reported_seq":1352,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:43.489064+0000","last_change":"2024-09-15T20:50:43.489064+0000","last_active":"2024-09-15T20:50:43.489064+0000","last_peered":"2024-09-15T20:50:43.489064+0000","last_clean":"2024-09-15T20:50:43.489064+0000","last_became_active":"2024-09-15T20:32:49.943293+0000","last_became_peered":"2024-09-15T20:32:49.943293+0000","last_unstale":"2024-09-15T20:50:43.489064+0000","last_undegraded":"2024-09-15T20:50:43.489064+0000","last_fullsized":"2024-09-15T20:50:43.489064+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:43.488845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:43.488845+0000","last_clean_scrub_stamp":"2024-09-15T20:50:43.488845+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:46:19.509126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:44.211770+0000","last_change":"2024-09-15T20:50:44.211770+0000","last_active":"2024-09-15T20:50:44.211770+0000","last_peered":"2024-09-15T20:50:44.211770+0000","last_clean":"2024-09-15T20:50:44.211770+0000","last_became_active":"2024-09-15T20:31:45.795437+0000","last_became_peered":"2024-09-15T20:31:45.795437+0000","last_unstale":"2024-09-15T20:50:44.211770+0000","last_undegraded":"2024-09-15T20:50:44.211770+0000","last_fullsized":"2024-09-15T20:50:44.211770+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:44.211578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:44.211578+0000","last_clean_scrub_stamp":"2024-09-15T20:50:44.211578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:50:25.291128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00084984299999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.b","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:43.577790+0000","last_change":"2024-09-15T20:50:43.577654+0000","last_active":"2024-09-15T20:50:43.577790+0000","last_peered":"2024-09-15T20:50:43.577790+0000","last_clean":"2024-09-15T20:50:43.577790+0000","last_became_active":"2024-09-15T20:32:49.926970+0000","last_became_peered":"2024-09-15T20:32:49.926970+0000","last_unstale":"2024-09-15T20:50:43.577790+0000","last_undegraded":"2024-09-15T20:50:43.577790+0000","last_fullsized":"2024-09-15T20:50:43.577790+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:43.577496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:43.577496+0000","last_clean_scrub_stamp":"2024-09-15T20:50:43.577496+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:07:39.567281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":1368,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:41.476819+0000","last_change":"2024-09-15T20:50:41.476819+0000","last_active":"2024-09-15T20:50:41.476819+0000","last_peered":"2024-09-15T20:50:41.476819+0000","last_clean":"2024-09-15T20:50:41.476819+0000","last_became_active":"2024-09-15T20:31:45.804668+0000","last_became_peered":"2024-09-15T20:31:45.804668+0000","last_unstale":"2024-09-15T20:50:41.476819+0000","last_undegraded":"2024-09-15T20:50:41.476819+0000","last_fullsized":"2024-09-15T20:50:41.476819+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:41.476730+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:41.476730+0000","last_clean_scrub_stamp":"2024-09-15T20:50:41.476730+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:46:59.912480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035281400000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.a","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:47.218325+0000","last_change":"2024-09-15T20:50:47.218325+0000","last_active":"2024-09-15T20:50:47.218325+0000","last_peered":"2024-09-15T20:50:47.218325+0000","last_clean":"2024-09-15T20:50:47.218325+0000","last_became_active":"2024-09-15T20:32:49.857975+0000","last_became_peered":"2024-09-15T20:32:49.857975+0000","last_unstale":"2024-09-15T20:50:47.218325+0000","last_undegraded":"2024-09-15T20:50:47.218325+0000","last_fullsized":"2024-09-15T20:50:47.218325+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:47.218169+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:47.218169+0000","last_clean_scrub_stamp":"2024-09-15T20:50:47.218169+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:02:41.810308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"703'3","reported_seq":1360,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:43.217077+0000","last_change":"2024-09-15T20:50:43.217077+0000","last_active":"2024-09-15T20:50:43.217077+0000","last_peered":"2024-09-15T20:50:43.217077+0000","last_clean":"2024-09-15T20:50:43.217077+0000","last_became_active":"2024-09-15T20:31:45.804107+0000","last_became_peered":"2024-09-15T20:31:45.804107+0000","last_unstale":"2024-09-15T20:50:43.217077+0000","last_undegraded":"2024-09-15T20:50:43.217077+0000","last_fullsized":"2024-09-15T20:50:43.217077+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"703'3","last_scrub_stamp":"2024-09-15T20:50:43.216914+0000","last_deep_scrub":"703'3","last_deep_scrub_stamp":"2024-09-15T20:50:43.216914+0000","last_clean_scrub_stamp":"2024-09-15T20:50:43.216914+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:15:41.953232+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.002401023,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.9","version":"0'0","reported_seq":1349,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:42.562037+0000","last_change":"2024-09-15T20:50:42.562037+0000","last_active":"2024-09-15T20:50:42.562037+0000","last_peered":"2024-09-15T20:50:42.562037+0000","last_clean":"2024-09-15T20:50:42.562037+0000","last_became_active":"2024-09-15T20:32:49.949390+0000","last_became_peered":"2024-09-15T20:32:49.949390+0000","last_unstale":"2024-09-15T20:50:42.562037+0000","last_undegraded":"2024-09-15T20:50:42.562037+0000","last_fullsized":"2024-09-15T20:50:42.562037+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:42.561949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:42.561949+0000","last_clean_scrub_stamp":"2024-09-15T20:50:42.561949+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:44:17.023668+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":1358,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:41.472767+0000","last_change":"2024-09-15T20:50:41.472767+0000","last_active":"2024-09-15T20:50:41.472767+0000","last_peered":"2024-09-15T20:50:41.472767+0000","last_clean":"2024-09-15T20:50:41.472767+0000","last_became_active":"2024-09-15T20:31:46.085143+0000","last_became_peered":"2024-09-15T20:31:46.085143+0000","last_unstale":"2024-09-15T20:50:41.472767+0000","last_undegraded":"2024-09-15T20:50:41.472767+0000","last_fullsized":"2024-09-15T20:50:41.472767+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:41.472667+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:41.472667+0000","last_clean_scrub_stamp":"2024-09-15T20:50:41.472667+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:10:58.267597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00091141699999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.f","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:47.242578+0000","last_change":"2024-09-15T20:50:47.242435+0000","last_active":"2024-09-15T20:50:47.242578+0000","last_peered":"2024-09-15T20:50:47.242578+0000","last_clean":"2024-09-15T20:50:47.242578+0000","last_became_active":"2024-09-15T20:32:49.909430+0000","last_became_peered":"2024-09-15T20:32:49.909430+0000","last_unstale":"2024-09-15T20:50:47.242578+0000","last_undegraded":"2024-09-15T20:50:47.242578+0000","last_fullsized":"2024-09-15T20:50:47.242578+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:47.242271+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:47.242271+0000","last_clean_scrub_stamp":"2024-09-15T20:50:47.242271+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:09:30.638474+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":1359,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:49.206309+0000","last_change":"2024-09-15T20:50:49.206309+0000","last_active":"2024-09-15T20:50:49.206309+0000","last_peered":"2024-09-15T20:50:49.206309+0000","last_clean":"2024-09-15T20:50:49.206309+0000","last_became_active":"2024-09-15T20:31:46.084925+0000","last_became_peered":"2024-09-15T20:31:46.084925+0000","last_unstale":"2024-09-15T20:50:49.206309+0000","last_undegraded":"2024-09-15T20:50:49.206309+0000","last_fullsized":"2024-09-15T20:50:49.206309+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:49.206120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:49.206120+0000","last_clean_scrub_stamp":"2024-09-15T20:50:49.206120+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T21:08:04.534542+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.00092452799999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.e","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:48.232608+0000","last_change":"2024-09-15T20:50:48.232439+0000","last_active":"2024-09-15T20:50:48.232608+0000","last_peered":"2024-09-15T20:50:48.232608+0000","last_clean":"2024-09-15T20:50:48.232608+0000","last_became_active":"2024-09-15T20:32:49.893603+0000","last_became_peered":"2024-09-15T20:32:49.893603+0000","last_unstale":"2024-09-15T20:50:48.232608+0000","last_undegraded":"2024-09-15T20:50:48.232608+0000","last_fullsized":"2024-09-15T20:50:48.232608+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:48.232277+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:48.232277+0000","last_clean_scrub_stamp":"2024-09-15T20:50:48.232277+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:23:56.772934+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":1357,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:43.535444+0000","last_change":"2024-09-15T20:50:43.535444+0000","last_active":"2024-09-15T20:50:43.535444+0000","last_peered":"2024-09-15T20:50:43.535444+0000","last_clean":"2024-09-15T20:50:43.535444+0000","last_became_active":"2024-09-15T20:31:45.795096+0000","last_became_peered":"2024-09-15T20:31:45.795096+0000","last_unstale":"2024-09-15T20:50:43.535444+0000","last_undegraded":"2024-09-15T20:50:43.535444+0000","last_fullsized":"2024-09-15T20:50:43.535444+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:43.535255+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:43.535255+0000","last_clean_scrub_stamp":"2024-09-15T20:50:43.535255+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:23:11.590005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0010592780000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.d","version":"0'0","reported_seq":1343,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:46.222766+0000","last_change":"2024-09-15T20:50:46.222600+0000","last_active":"2024-09-15T20:50:46.222766+0000","last_peered":"2024-09-15T20:50:46.222766+0000","last_clean":"2024-09-15T20:50:46.222766+0000","last_became_active":"2024-09-15T20:32:49.908352+0000","last_became_peered":"2024-09-15T20:32:49.908352+0000","last_unstale":"2024-09-15T20:50:46.222766+0000","last_undegraded":"2024-09-15T20:50:46.222766+0000","last_fullsized":"2024-09-15T20:50:46.222766+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:46.222444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:46.222444+0000","last_clean_scrub_stamp":"2024-09-15T20:50:46.222444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:39:04.943238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":1358,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:45.204239+0000","last_change":"2024-09-15T20:50:45.204239+0000","last_active":"2024-09-15T20:50:45.204239+0000","last_peered":"2024-09-15T20:50:45.204239+0000","last_clean":"2024-09-15T20:50:45.204239+0000","last_became_active":"2024-09-15T20:31:45.975014+0000","last_became_peered":"2024-09-15T20:31:45.975014+0000","last_unstale":"2024-09-15T20:50:45.204239+0000","last_undegraded":"2024-09-15T20:50:45.204239+0000","last_fullsized":"2024-09-15T20:50:45.204239+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:45.204091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:45.204091+0000","last_clean_scrub_stamp":"2024-09-15T20:50:45.204091+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:55:13.699400+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.001313996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.c","version":"0'0","reported_seq":1352,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:42.455719+0000","last_change":"2024-09-15T20:50:42.455667+0000","last_active":"2024-09-15T20:50:42.455719+0000","last_peered":"2024-09-15T20:50:42.455719+0000","last_clean":"2024-09-15T20:50:42.455719+0000","last_became_active":"2024-09-15T20:32:49.969795+0000","last_became_peered":"2024-09-15T20:32:49.969795+0000","last_unstale":"2024-09-15T20:50:42.455719+0000","last_undegraded":"2024-09-15T20:50:42.455719+0000","last_fullsized":"2024-09-15T20:50:42.455719+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:42.455606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:42.455606+0000","last_clean_scrub_stamp":"2024-09-15T20:50:42.455606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T08:46:10.657482+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"9.1e","version":"0'0","reported_seq":1350,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:47.531010+0000","last_change":"2024-09-15T20:50:47.530885+0000","last_active":"2024-09-15T20:50:47.531010+0000","last_peered":"2024-09-15T20:50:47.531010+0000","last_clean":"2024-09-15T20:50:47.531010+0000","last_became_active":"2024-09-15T20:32:49.948678+0000","last_became_peered":"2024-09-15T20:32:49.948678+0000","last_unstale":"2024-09-15T20:50:47.531010+0000","last_undegraded":"2024-09-15T20:50:47.531010+0000","last_fullsized":"2024-09-15T20:50:47.531010+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:47.530719+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:47.530719+0000","last_clean_scrub_stamp":"2024-09-15T20:50:47.530719+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T04:40:51.878495+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1f","version":"0'0","reported_seq":1342,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:47.600093+0000","last_change":"2024-09-15T20:50:47.600093+0000","last_active":"2024-09-15T20:50:47.600093+0000","last_peered":"2024-09-15T20:50:47.600093+0000","last_clean":"2024-09-15T20:50:47.600093+0000","last_became_active":"2024-09-15T20:32:49.872991+0000","last_became_peered":"2024-09-15T20:32:49.872991+0000","last_unstale":"2024-09-15T20:50:47.600093+0000","last_undegraded":"2024-09-15T20:50:47.600093+0000","last_fullsized":"2024-09-15T20:50:47.600093+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:47.599938+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:47.599938+0000","last_clean_scrub_stamp":"2024-09-15T20:50:47.599938+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:55:23.542200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.1c","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:46.521880+0000","last_change":"2024-09-15T20:50:46.521880+0000","last_active":"2024-09-15T20:50:46.521880+0000","last_peered":"2024-09-15T20:50:46.521880+0000","last_clean":"2024-09-15T20:50:46.521880+0000","last_became_active":"2024-09-15T20:32:50.021808+0000","last_became_peered":"2024-09-15T20:32:50.021808+0000","last_unstale":"2024-09-15T20:50:46.521880+0000","last_undegraded":"2024-09-15T20:50:46.521880+0000","last_fullsized":"2024-09-15T20:50:46.521880+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:46.521718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:46.521718+0000","last_clean_scrub_stamp":"2024-09-15T20:50:46.521718+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T23:43:58.025890+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.1d","version":"0'0","reported_seq":1348,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:44.532820+0000","last_change":"2024-09-15T20:50:44.532820+0000","last_active":"2024-09-15T20:50:44.532820+0000","last_peered":"2024-09-15T20:50:44.532820+0000","last_clean":"2024-09-15T20:50:44.532820+0000","last_became_active":"2024-09-15T20:32:49.937453+0000","last_became_peered":"2024-09-15T20:32:49.937453+0000","last_unstale":"2024-09-15T20:50:44.532820+0000","last_undegraded":"2024-09-15T20:50:44.532820+0000","last_fullsized":"2024-09-15T20:50:44.532820+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:44.532646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:44.532646+0000","last_clean_scrub_stamp":"2024-09-15T20:50:44.532646+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:01:51.286247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"9.12","version":"0'0","reported_seq":1340,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:48.237455+0000","last_change":"2024-09-15T20:50:48.237455+0000","last_active":"2024-09-15T20:50:48.237455+0000","last_peered":"2024-09-15T20:50:48.237455+0000","last_clean":"2024-09-15T20:50:48.237455+0000","last_became_active":"2024-09-15T20:32:49.906485+0000","last_became_peered":"2024-09-15T20:32:49.906485+0000","last_unstale":"2024-09-15T20:50:48.237455+0000","last_undegraded":"2024-09-15T20:50:48.237455+0000","last_fullsized":"2024-09-15T20:50:48.237455+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:48.237296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:48.237296+0000","last_clean_scrub_stamp":"2024-09-15T20:50:48.237296+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T00:46:58.653199+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"9.13","version":"0'0","reported_seq":1342,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:51.230130+0000","last_change":"2024-09-15T20:50:51.230130+0000","last_active":"2024-09-15T20:50:51.230130+0000","last_peered":"2024-09-15T20:50:51.230130+0000","last_clean":"2024-09-15T20:50:51.230130+0000","last_became_active":"2024-09-15T20:32:49.844315+0000","last_became_peered":"2024-09-15T20:32:49.844315+0000","last_unstale":"2024-09-15T20:50:51.230130+0000","last_undegraded":"2024-09-15T20:50:51.230130+0000","last_fullsized":"2024-09-15T20:50:51.230130+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:51.229964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:51.229964+0000","last_clean_scrub_stamp":"2024-09-15T20:50:51.229964+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:00:29.271110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.10","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:44.064358+0000","last_change":"2024-09-15T20:50:44.064358+0000","last_active":"2024-09-15T20:50:44.064358+0000","last_peered":"2024-09-15T20:50:44.064358+0000","last_clean":"2024-09-15T20:50:44.064358+0000","last_became_active":"2024-09-15T20:32:49.930427+0000","last_became_peered":"2024-09-15T20:32:49.930427+0000","last_unstale":"2024-09-15T20:50:44.064358+0000","last_undegraded":"2024-09-15T20:50:44.064358+0000","last_fullsized":"2024-09-15T20:50:44.064358+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:44.064089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:44.064089+0000","last_clean_scrub_stamp":"2024-09-15T20:50:44.064089+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T01:07:06.446342+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.11","version":"0'0","reported_seq":1342,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:50.239953+0000","last_change":"2024-09-15T20:50:50.239953+0000","last_active":"2024-09-15T20:50:50.239953+0000","last_peered":"2024-09-15T20:50:50.239953+0000","last_clean":"2024-09-15T20:50:50.239953+0000","last_became_active":"2024-09-15T20:32:49.880025+0000","last_became_peered":"2024-09-15T20:32:49.880025+0000","last_unstale":"2024-09-15T20:50:50.239953+0000","last_undegraded":"2024-09-15T20:50:50.239953+0000","last_fullsized":"2024-09-15T20:50:50.239953+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:50.239745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:50.239745+0000","last_clean_scrub_stamp":"2024-09-15T20:50:50.239745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:20:46.706649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2],"acting":[3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.16","version":"0'0","reported_seq":1335,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:45.084739+0000","last_change":"2024-09-15T20:50:45.084739+0000","last_active":"2024-09-15T20:50:45.084739+0000","last_peered":"2024-09-15T20:50:45.084739+0000","last_clean":"2024-09-15T20:50:45.084739+0000","last_became_active":"2024-09-15T20:32:49.939238+0000","last_became_peered":"2024-09-15T20:32:49.939238+0000","last_unstale":"2024-09-15T20:50:45.084739+0000","last_undegraded":"2024-09-15T20:50:45.084739+0000","last_fullsized":"2024-09-15T20:50:45.084739+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:45.084578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:45.084578+0000","last_clean_scrub_stamp":"2024-09-15T20:50:45.084578+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T06:05:32.168295+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"9.17","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:52.208530+0000","last_change":"2024-09-15T20:50:52.208530+0000","last_active":"2024-09-15T20:50:52.208530+0000","last_peered":"2024-09-15T20:50:52.208530+0000","last_clean":"2024-09-15T20:50:52.208530+0000","last_became_active":"2024-09-15T20:32:49.903971+0000","last_became_peered":"2024-09-15T20:32:49.903971+0000","last_unstale":"2024-09-15T20:50:52.208530+0000","last_undegraded":"2024-09-15T20:50:52.208530+0000","last_fullsized":"2024-09-15T20:50:52.208530+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:52.208355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:52.208355+0000","last_clean_scrub_stamp":"2024-09-15T20:50:52.208355+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T07:43:04.193041+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"9.14","version":"0'0","reported_seq":1341,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:46.627949+0000","last_change":"2024-09-15T20:50:46.627949+0000","last_active":"2024-09-15T20:50:46.627949+0000","last_peered":"2024-09-15T20:50:46.627949+0000","last_clean":"2024-09-15T20:50:46.627949+0000","last_became_active":"2024-09-15T20:32:49.909018+0000","last_became_peered":"2024-09-15T20:32:49.909018+0000","last_unstale":"2024-09-15T20:50:46.627949+0000","last_undegraded":"2024-09-15T20:50:46.627949+0000","last_fullsized":"2024-09-15T20:50:46.627949+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:46.627791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:46.627791+0000","last_clean_scrub_stamp":"2024-09-15T20:50:46.627791+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-17T03:45:44.198229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"9.15","version":"0'0","reported_seq":1334,"reported_epoch":703,"state":"active+clean","last_fresh":"2024-09-15T20:50:43.097416+0000","last_change":"2024-09-15T20:50:43.097416+0000","last_active":"2024-09-15T20:50:43.097416+0000","last_peered":"2024-09-15T20:50:43.097416+0000","last_clean":"2024-09-15T20:50:43.097416+0000","last_became_active":"2024-09-15T20:32:49.942857+0000","last_became_peered":"2024-09-15T20:32:49.942857+0000","last_unstale":"2024-09-15T20:50:43.097416+0000","last_undegraded":"2024-09-15T20:50:43.097416+0000","last_fullsized":"2024-09-15T20:50:43.097416+0000","mapping_epoch":21,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-15T20:50:43.097258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-15T20:50:43.097258+0000","last_clean_scrub_stamp":"2024-09-15T20:50:43.097258+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-16T22:12:27.793308+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":9,"num_pg":32,"stat_sum":{"num_bytes":5,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":10,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":64,"acting":64,"num_store_stats":6},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":6},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":1057856,"data_compressed":61070,"data_compressed_allocated":81920,"data_compressed_original":1024000,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":5,"up_from":12,"seq":51539607787,"num_pgs":12,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":12,"seq":51539607786,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72536,"kb_used_data":5256,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299304,"statfs":{"total":96636764160,"available":96562487296,"internally_reserved":0,"allocated":5382144,"data_stored":2198026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":12,"seq":51539607787,"num_pgs":11,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":50332,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":94321508,"statfs":{"total":96636764160,"available":96585224192,"internally_reserved":0,"allocated":5386240,"data_stored":2198031,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":12,"seq":51539607786,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72600,"kb_used_data":5320,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299240,"statfs":{"total":96636764160,"available":96562421760,"internally_reserved":0,"allocated":5447680,"data_stored":2726959,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":11,"seq":47244640492,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":68436,"kb_used_data":5316,"kb_used_omap":1,"kb_used_meta":63102,"kb_avail":94303404,"statfs":{"total":96636764160,"available":96566685696,"internally_reserved":0,"allocated":5443584,"data_stored":2726954,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":1593,"internal_metadata":64616903},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":11,"seq":47244640492,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72540,"kb_used_data":5260,"kb_used_omap":1,"kb_used_meta":67262,"kb_avail":94299300,"statfs":{"total":96636764160,"available":96562483200,"internally_reserved":0,"allocated":5386240,"data_stored":2198045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":68876741},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":528928,"data_compressed":30535,"data_compressed_allocated":40960,"data_compressed_original":512000,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":5,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":9,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-15T20:51:03.243 DEBUG:teuthology.orchestra.run.smithi078:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-09-15T20:51:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.348+0000 7f3d3952f640 10 monclient: get_monmap_and_config 2024-09-15T20:51:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.348+0000 7f3d3952f640 10 monclient: build_initial_monmap 2024-09-15T20:51:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.348+0000 7f3d3952f640 10 monclient: monmap: 2024-09-15T20:51:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:51:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:51:03.352221+0000 2024-09-15T20:51:03.350 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:51:03.352221+0000 2024-09-15T20:51:03.351 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:51:03.351 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.351 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:51:03.351 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-c 2024-09-15T20:51:03.351 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-b 2024-09-15T20:51:03.351 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 Processor -- start 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- start start 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: init 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: _reopen_session rank -1 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: _add_conns ranks=[2,1,0] 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): picked mon.noname-b con 0x7f3d34137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): picked mon.noname-c con 0x7f3d34074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3413e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:51:03.358 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d34138850 con 0x7f3d3413e370 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d340a5da0 con 0x7f3d34074500 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d34073560 con 0x7f3d34137d60 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): _renew_subs 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): authenticate will time out at 2324.236769s 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d32ffd640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f3d34137d60 0x7f3d34138140 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56666/0 (socket says 172.21.15.78:56666) 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d32ffd640 1 -- 172.21.15.78:0/244637475 learned_addr learned my addr 172.21.15.78:0/244637475 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d337fe640 1 --1- 172.21.15.78:0/244637475 >> v1:172.21.15.78:6789/0 conn(0x7f3d3413e370 0x7f3d34140760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.78:6789/0 says I am v1:172.21.15.78:45076/0 (socket says 172.21.15.78:45076) 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3653854351 0 0) 0x7f3d34073560 con 0x7f3d34137d60 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient(hunting): my global_id is 154340 2024-09-15T20:51:03.359 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d20003530 con 0x7f3d34137d60 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3724812286 0 0) 0x7f3d34138850 con 0x7f3d3413e370 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient(hunting): my global_id is 144420 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d34073560 con 0x7f3d3413e370 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 467742068 0 0) 0x7f3d20003530 con 0x7f3d34137d60 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 >> v1:172.21.15.146:6789/0 conn(0x7f3d34074500 legacy=0x7f3d3413cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 >> v1:172.21.15.78:6789/0 conn(0x7f3d3413e370 legacy=0x7f3d34140760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: _finish_hunting 0 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 monclient: found mon.noname-b 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:51:03.360 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.78:6790/0 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d34073840 con 0x7f3d34137d60 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: _finish_auth 0 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: _check_auth_tickets 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3d240016a0 con 0x7f3d34137d60 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: dump: 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.361 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 5 monclient: authenticate success, global_id 154340 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 <== mon.2 v1:172.21.15.78:6790/0 4 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d240018c0 con 0x7f3d34137d60 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: handle_config config(0 keys) 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 1 -- 172.21.15.78:0/244637475 <== mon.2 v1:172.21.15.78:6790/0 5 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3d24001cd0 con 0x7f3d34137d60 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:51:03.362 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d327fc640 10 monclient: dump: 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.363 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: get_monmap_and_config success 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 4 set_mon_vals no callback set 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: shutdown 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- 172.21.15.78:0/244637475 >> v1:172.21.15.78:6790/0 conn(0x7f3d34137d60 legacy=0x7f3d34138140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- 172.21.15.78:0/244637475 shutdown_connections 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- 172.21.15.78:0/244637475 >> 172.21.15.78:0/244637475 conn(0x7f3d340aa650 msgr2=0x7f3d340aaa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- 172.21.15.78:0/244637475 shutdown_connections 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- 172.21.15.78:0/244637475 wait complete. 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: build_initial_monmap 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: monmap: 2024-09-15T20:51:03.364 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 0 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 0.000000 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:created 0.000000 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 0 (unknown) 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.noname-a 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.noname-b 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.noname-c 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 librados: starting msgr at 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 librados: starting objecter 2024-09-15T20:51:03.365 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 Processor -- start 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- start start 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 librados: setting wanted keys 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 librados: calling monclient init 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: init 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: _reopen_session rank -1 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient: _add_conns ranks=[1,2,0] 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): picked mon.noname-b con 0x7f3d34074500 addr v1:172.21.15.146:6789/0 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): picked mon.noname-c con 0x7f3d34137d60 addr v1:172.21.15.78:6790/0 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): picked mon.noname-a con 0x7f3d3413e370 addr v1:172.21.15.78:6789/0 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- --> v1:172.21.15.78:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d3409f500 con 0x7f3d3413e370 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- --> v1:172.21.15.146:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d34141780 con 0x7f3d34074500 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 1 -- --> v1:172.21.15.78:6790/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d34132e20 con 0x7f3d34137d60 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): _renew_subs 2024-09-15T20:51:03.366 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d3952f640 10 monclient(hunting): authenticate will time out at 2324.239600s 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2a7fc640 1 --1- >> v1:172.21.15.78:6790/0 conn(0x7f3d34137d60 0x7f3d3409edf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.78:6790/0 says I am v1:172.21.15.78:56674/0 (socket says 172.21.15.78:56674) 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2a7fc640 1 -- 172.21.15.78:0/3355016210 learned_addr learned my addr 172.21.15.78:0/3355016210 (peer_addr_for_me v1:172.21.15.78:0/0) 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 client.?.objecter ms_handle_connect 0x7f3d34137d60 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 client.?.objecter ms_handle_connect 0x7f3d3413e370 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2596580080 0 0) 0x7f3d34132e20 con 0x7f3d34137d60 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 monclient(hunting): my global_id is 154343 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6790/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d14004ec0 con 0x7f3d34137d60 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.0 v1:172.21.15.78:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1571467693 0 0) 0x7f3d3409f500 con 0x7f3d3413e370 2024-09-15T20:51:03.367 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 10 monclient(hunting): my global_id is 144423 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.352+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d34132e20 con 0x7f3d3413e370 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 client.?.objecter ms_handle_connect 0x7f3d34074500 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 client.?.objecter resend_mon_ops 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 3257613154 0 0) 0x7f3d14004ec0 con 0x7f3d34137d60 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6790/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d3409f500 con 0x7f3d34137d60 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3d1c001a40 con 0x7f3d34137d60 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): dump: 2024-09-15T20:51:03.368 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): _finish_auth 0 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.0 v1:172.21.15.78:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 764+0+0 (unknown 4269798060 0 0) 0x7f3d34132e20 con 0x7f3d3413e370 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6789/0 -- auth(proto 2 165 bytes epoch 0) -- 0x7f3d14004ec0 con 0x7f3d3413e370 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.0 v1:172.21.15.78:6789/0 3 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3d2c0013f0 con 0x7f3d3413e370 2024-09-15T20:51:03.369 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): dump: 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.1 v1:172.21.15.146:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1838620170 0 0) 0x7f3d34141780 con 0x7f3d34074500 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): _negotiate 2024-09-15T20:51:03.370 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): _init_auth method 2 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): _init_auth creating new auth 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient(hunting): my global_id is 164236 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.146:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d34132e20 con 0x7f3d34074500 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (unknown 1580541420 0 0) 0x7f3d3409f500 con 0x7f3d34137d60 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 >> v1:172.21.15.146:6789/0 conn(0x7f3d34074500 legacy=0x7f3d3409e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 >> v1:172.21.15.78:6789/0 conn(0x7f3d3413e370 legacy=0x7f3d341d4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient: _finish_hunting 0 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 monclient: found mon.c 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6790/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d34132580 con 0x7f3d34137d60 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient: _finish_auth 0 2024-09-15T20:51:03.371 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient: _check_auth_tickets 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 5 monclient: authenticate success, global_id 154343 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 10 monclient: _renew_subs 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6790/0 -- mon_subscribe({mgrmap=0+}) -- 0x7f3d3409ff10 con 0x7f3d34137d60 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 10 client.154343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 10 monclient: _renew_subs 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6790/0 -- mon_subscribe({osdmap=0}) -- 0x7f3d341374b0 con 0x7f3d34137d60 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 1 librados: init done 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 5 ==== config(0 keys) ==== 4+0+0 (unknown 0 0 0) 0x7f3d1c001d40 con 0x7f3d34137d60 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient: handle_config config(0 keys) 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 6 ==== mon_map magic: 0 ==== 308+0+0 (unknown 3703550162 0 0) 0x7f3d1c003a70 con 0x7f3d34137d60 2024-09-15T20:51:03.372 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient: handle_monmap mon_map magic: 0 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d31ffb640 4 set_mon_vals no callback set 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 monclient: dump: 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:epoch 1 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:fsid a8a3eb37-3a92-4568-8aad-3536499064e4 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:last_changed 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:created 2024-09-15T20:31:01.119418+0000 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:min_mon_release 19 (squid) 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:election_strategy: 1 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:0: v1:172.21.15.78:6789/0 mon.a 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:1: v1:172.21.15.146:6789/0 mon.b 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:2: v1:172.21.15.78:6790/0 mon.c 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:03.373 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 7 ==== mgrmap(e 5) ==== 48832+0+0 (unknown 231138899 0 0) 0x7f3d1c003c10 con 0x7f3d34137d60 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d18005090 con 0x7f3d34137d60 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 8 ==== osd_map(703..703 src has 379..703) ==== 4673+0+0 (unknown 3420109922 0 0) 0x7f3d1c04b5a0 con 0x7f3d34137d60 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 client.154343.objecter ms_dispatch 0x7f3d34000ba0 osd_map(703..703 src has 379..703) 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 3 client.154343.objecter handle_osd_map got epochs [703,703] > 0 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 3 client.154343.objecter handle_osd_map decoding full epoch 703 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 20 client.154343.objecter dump_active .. 0 homeless 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d32ffd640 10 monclient: get_auth_request con 0x7f3d1403e2b0 auth_method 0 2024-09-15T20:51:03.374 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.356+0000 7f3d2bfff640 10 client.154343.objecter ms_handle_connect 0x7f3d1403e2b0 2024-09-15T20:51:03.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.360+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+187702 (unknown 1092875540 0 2948034943) 0x7f3d1c016b00 con 0x7f3d34137d60 2024-09-15T20:51:03.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.360+0000 7f3d2bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-09-15T20:51:03.375 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.360+0000 7f3d2bfff640 10 monclient: _finish_command 1 = system:0 2024-09-15T20:51:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.532+0000 7f3d3952f640 10 start_mon_command cmd=[{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T20:51:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.532+0000 7f3d3952f640 10 monclient: _send_command 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T20:51:03.533 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.532+0000 7f3d3952f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.78:6790/0 2024-09-15T20:51:03.534 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.532+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 --> v1:172.21.15.78:6790/0 -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7f3d18005380 con 0x7f3d34137d60 2024-09-15T20:51:03.543 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.544+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 10 ==== config(1 keys) ==== 35+0+0 (unknown 2555425225 0 0) 0x7f3d1c015f70 con 0x7f3d34137d60 2024-09-15T20:51:03.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.544+0000 7f3d2bfff640 10 monclient: handle_config config(1 keys) 2024-09-15T20:51:03.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.544+0000 7f3d31ffb640 4 set_mon_vals no callback set 2024-09-15T20:51:03.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.544+0000 7f3d31ffb640 10 set_mon_vals mon_health_to_clog = false 2024-09-15T20:51:03.544 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.544+0000 7f3d2bfff640 1 -- 172.21.15.78:0/3355016210 <== mon.2 v1:172.21.15.78:6790/0 11 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v6)=0 v6) ==== 123+0+0 (unknown 1453024670 0 0) 0x7f3d1c016400 con 0x7f3d34137d60 2024-09-15T20:51:03.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.544+0000 7f3d2bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "config set", "who": "global", "name": "mon_health_to_clog", "value": "false"}] 2024-09-15T20:51:03.545 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.544+0000 7f3d2bfff640 10 monclient: _finish_command 2 = system:0 2024-09-15T20:51:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 10 librados: watch_flush enter 2024-09-15T20:51:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 10 librados: watch_flush exit 2024-09-15T20:51:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 20 client.154343.objecter shutdown clearing up homeless session... 2024-09-15T20:51:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 10 client.154343.objecter successfully canceled tick 2024-09-15T20:51:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 >> v1:172.21.15.146:6812/435598642 conn(0x7f3d1403e2b0 legacy=0x7f3d14040770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.548 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 10 monclient: shutdown 2024-09-15T20:51:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 20 monclient: shutdown discarding 0 pending message(s) 2024-09-15T20:51:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 >> v1:172.21.15.78:6790/0 conn(0x7f3d34137d60 legacy=0x7f3d3409edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-09-15T20:51:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 shutdown_connections 2024-09-15T20:51:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 >> 172.21.15.78:0/3355016210 conn(0x7f3d340aa650 msgr2=0x7f3d340a8150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-09-15T20:51:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 shutdown_connections 2024-09-15T20:51:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 1 -- 172.21.15.78:0/3355016210 wait complete. 2024-09-15T20:51:03.549 INFO:teuthology.orchestra.run.smithi078.stderr:2024-09-15T20:51:03.548+0000 7f3d3952f640 1 librados: shutdown 2024-09-15T20:51:03.563 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-15T20:51:03.563 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-15T20:51:03.563 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-09-15T20:51:03.563 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:03.645 INFO:tasks.ceph.osd.0:Stopped 2024-09-15T20:51:03.645 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-09-15T20:51:03.646 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:03.725 INFO:tasks.ceph.osd.1:Stopped 2024-09-15T20:51:03.725 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-09-15T20:51:03.725 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:03.808 INFO:tasks.ceph.osd.2:Stopped 2024-09-15T20:51:03.808 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-09-15T20:51:03.808 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:03.886 INFO:tasks.ceph.osd.3:Stopped 2024-09-15T20:51:03.886 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-09-15T20:51:03.886 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:03.981 INFO:tasks.ceph.osd.4:Stopped 2024-09-15T20:51:03.982 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-09-15T20:51:03.982 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:04.078 INFO:tasks.ceph.osd.5:Stopped 2024-09-15T20:51:04.078 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-15T20:51:04.078 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-09-15T20:51:04.078 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:04.113 INFO:tasks.ceph.mgr.x:Stopped 2024-09-15T20:51:04.114 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-15T20:51:04.114 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-09-15T20:51:04.114 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:04.141 INFO:tasks.ceph.mon.a:Stopped 2024-09-15T20:51:04.141 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-09-15T20:51:04.141 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:04.203 INFO:tasks.ceph.mon.c:Stopped 2024-09-15T20:51:04.203 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-09-15T20:51:04.203 INFO:teuthology.orchestra.run:waiting for 300 2024-09-15T20:51:04.221 INFO:tasks.ceph.mon.b:Stopped 2024-09-15T20:51:04.221 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-15T20:51:04.221 DEBUG:teuthology.orchestra.run.smithi078:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v ScrubResult | egrep -v 'scrub mismatch' | egrep -v 'overall HEALTH_' | egrep -v '\(MGR_DOWN\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(PG_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v 'slow request' | head -n 1 2024-09-15T20:51:04.265 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:51:04.265 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-09-15T20:51:04.679 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:51:04.679 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-09-15T20:51:04.843 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:51:04.843 DEBUG:teuthology.orchestra.run.smithi078:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-09-15T20:51:04.978 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:51:04.978 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-09-15T20:51:05.255 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:51:05.256 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-09-15T20:51:05.467 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:51:05.468 DEBUG:teuthology.orchestra.run.smithi146:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-09-15T20:51:05.666 INFO:tasks.ceph:Archiving mon data... 2024-09-15T20:51:05.667 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/data/mon.a.tgz 2024-09-15T20:51:05.667 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-09-15T20:51:05.673 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.wgxkJyXyJ6 2024-09-15T20:51:05.673 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.wgxkJyXyJ6 2024-09-15T20:51:05.801 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.wgxkJyXyJ6 2024-09-15T20:51:05.864 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.wgxkJyXyJ6 is 421KB 2024-09-15T20:51:05.925 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.wgxkJyXyJ6 2024-09-15T20:51:05.931 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/data/mon.c.tgz 2024-09-15T20:51:05.931 DEBUG:teuthology.orchestra.run.smithi078:> mktemp 2024-09-15T20:51:05.983 INFO:teuthology.orchestra.run.smithi078.stdout:/tmp/tmp.5FCOHjgpVh 2024-09-15T20:51:05.984 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.5FCOHjgpVh 2024-09-15T20:51:06.111 DEBUG:teuthology.orchestra.run.smithi078:> sudo chmod 0666 /tmp/tmp.5FCOHjgpVh 2024-09-15T20:51:06.175 DEBUG:teuthology.orchestra.remote:smithi078:/tmp/tmp.5FCOHjgpVh is 419KB 2024-09-15T20:51:06.237 DEBUG:teuthology.orchestra.run.smithi078:> rm -fr /tmp/tmp.5FCOHjgpVh 2024-09-15T20:51:06.242 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/data/mon.b.tgz 2024-09-15T20:51:06.242 DEBUG:teuthology.orchestra.run.smithi146:> mktemp 2024-09-15T20:51:06.249 INFO:teuthology.orchestra.run.smithi146.stdout:/tmp/tmp.ldFGefBuBQ 2024-09-15T20:51:06.249 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.ldFGefBuBQ 2024-09-15T20:51:06.370 DEBUG:teuthology.orchestra.run.smithi146:> sudo chmod 0666 /tmp/tmp.ldFGefBuBQ 2024-09-15T20:51:06.435 DEBUG:teuthology.orchestra.remote:smithi146:/tmp/tmp.ldFGefBuBQ is 416KB 2024-09-15T20:51:06.497 DEBUG:teuthology.orchestra.run.smithi146:> rm -fr /tmp/tmp.ldFGefBuBQ 2024-09-15T20:51:06.503 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-15T20:51:06.503 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T20:51:06.506 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-09-15T20:51:06.567 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-15T20:51:06.607 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-15T20:51:06.608 INFO:tasks.ceph:Archiving crash dumps... 2024-09-15T20:51:06.609 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/remote/smithi078/crash 2024-09-15T20:51:06.610 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T20:51:06.624 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/remote/smithi146/crash 2024-09-15T20:51:06.625 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-15T20:51:06.667 INFO:tasks.ceph:Compressing logs... 2024-09-15T20:51:06.667 DEBUG:teuthology.orchestra.run.smithi078:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T20:51:06.672 DEBUG:teuthology.orchestra.run.smithi146:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-15T20:51:06.689 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28834.log 2024-09-15T20:51:06.690 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30319.log 2024-09-15T20:51:06.690 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28834.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31035.log 2024-09-15T20:51:06.691 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30319.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28834.log.gz 2024-09-15T20:51:06.691 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30806.log 2024-09-15T20:51:06.691 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29711.log 2024-09-15T20:51:06.691 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30319.log.gz 2024-09-15T20:51:06.691 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30806.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33437.log 2024-09-15T20:51:06.691 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31035.log.gz 2024-09-15T20:51:06.693 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30806.log.gz 2024-09-15T20:51:06.693 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29711.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28451.log 2024-09-15T20:51:06.693 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30903.log 2024-09-15T20:51:06.693 INFO:teuthology.orchestra.run.smithi078.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.33437.log.gz 2024-09-15T20:51:06.693 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29711.log.gz 2024-09-15T20:51:06.693 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33801.log 2024-09-15T20:51:06.693 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30903.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32459.log 2024-09-15T20:51:06.693 INFO:teuthology.orchestra.run.smithi078.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.28451.log.gz 2024-09-15T20:51:06.695 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33801.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28628.log 2024-09-15T20:51:06.695 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30903.log.gz 2024-09-15T20:51:06.695 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32459.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.33801.log.gz 2024-09-15T20:51:06.695 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28255.log 2024-09-15T20:51:06.695 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.32459.log.gz 2024-09-15T20:51:06.695 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28628.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-09-15T20:51:06.695 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28255.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28394.log 2024-09-15T20:51:06.696 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28628.log.gz 2024-09-15T20:51:06.696 INFO:teuthology.orchestra.run.smithi078.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.28255.log.gz 2024-09-15T20:51:06.696 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31191.log 2024-09-15T20:51:06.696 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25242.log 2024-09-15T20:51:06.698 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31191.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28394.log.gz 2024-09-15T20:51:06.698 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32064.log 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.25242.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28087.log 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31191.log.gz 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32064.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.25242.log.gz 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30137.log 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32064.log.gz 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33827.log 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28087.log: /var/log/ceph/ceph-client.admin.30137.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29630.log 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33827.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28087.log.gz 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30137.log.gz 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32248.log 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33827.log.gz 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28112.log 2024-09-15T20:51:06.699 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28253.log 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32248.log: /var/log/ceph/ceph-client.admin.28112.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33333.log 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.29630.log.gz 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28253.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33567.log 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.32248.log.gz 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.28112.log.gz 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28253.log.gz 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33333.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30544.log 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32792.log 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33333.log.gz 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30544.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31829.log 2024-09-15T20:51:06.703 INFO:teuthology.orchestra.run.smithi078.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33567.log.gz 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30544.log.gz 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28938.log 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32792.log: /var/log/ceph/ceph-client.admin.31829.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31061.log 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30728.log 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32792.log.gz 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31829.log.gz 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31061.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.28938.log.gz 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28210.log 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30728.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31009.log 2024-09-15T20:51:06.704 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31061.log.gz 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28210.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30728.log.gz 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.28210.log.gz 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30293.log 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.31009.log.gz 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28132.log 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.30293.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28158.log 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28132.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30293.log.gz 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31777.log 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28132.log.gz 2024-09-15T20:51:06.709 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32144.log 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28158.log: /var/log/ceph/ceph-client.admin.31777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29505.log 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32144.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31777.log.gz 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28158.log.gz 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29505.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32378.log 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32144.log.gz 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.a.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.29505.log.gz 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32661.log 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32378.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31139.log 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32661.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28860.log 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.32378.log.gz 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32661.log.gz 2024-09-15T20:51:06.710 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31139.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32900.log 2024-09-15T20:51:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30929.log 2024-09-15T20:51:06.711 INFO:teuthology.orchestra.run.smithi078.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.31139.log.gz 2024-09-15T20:51:06.711 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32900.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31610.log 2024-09-15T20:51:06.711 INFO:teuthology.orchestra.run.smithi078.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.28860.log.gz 2024-09-15T20:51:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30929.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.32900.log.gz 2024-09-15T20:51:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33140.log 2024-09-15T20:51:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31528.log 2024-09-15T20:51:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31610.log: /var/log/ceph/ceph-client.admin.33140.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30929.log.gz 2024-09-15T20:51:06.715 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33775.log 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31610.log.gz 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.33140.log.gz 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31217.log 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28256.log 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33775.log: /var/log/ceph/ceph-client.admin.31217.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.31528.log.gz 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32489.log 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33775.log.gz 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29877.log 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28256.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31217.log.gz 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33515.log 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28256.log.gz 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29877.log: gzip -5 86.5% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.32489.log.gz /var/log/ceph/ceph-client.admin.32978.log 2024-09-15T20:51:06.716 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32038.log 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33515.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29877.log.gz 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32978.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31580.log 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32038.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33515.log.gz 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31366.log 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32978.log.gz 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30596.log 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.32038.log.gz 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31366.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.31580.log.gz 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32118.log 2024-09-15T20:51:06.717 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28005.log 2024-09-15T20:51:06.718 INFO:teuthology.orchestra.run.smithi078.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.31366.log.gz 2024-09-15T20:51:06.718 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28964.log 2024-09-15T20:51:06.718 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30596.log.gz 2024-09-15T20:51:06.724 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28005.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30189.log 2024-09-15T20:51:06.724 INFO:teuthology.orchestra.run.smithi078.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.32118.log.gz 2024-09-15T20:51:06.724 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28964.log: /var/log/ceph/ceph-client.admin.29578.log 2024-09-15T20:51:06.724 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28005.log.gz 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30189.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33084.log 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.28964.log.gz 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30676.log 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30189.log.gz 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33084.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33281.log 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.29578.log.gz 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30241.log 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33084.log.gz 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33281.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28450.log 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30241.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.33281.log.gz 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr: 88.7%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29604.log 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30676.log.gz 2024-09-15T20:51:06.725 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28450.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33671.log 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30241.log.gz 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29604.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29737.log 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28450.log.gz 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.29604.log.gz 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33541.log 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.29737.log: /var/log/ceph/ceph-client.admin.31554.log 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33671.log.gz 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33541.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30449.log 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.29737.log.gz 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31554.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30478.log 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33541.log.gz 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33110.log 2024-09-15T20:51:06.726 INFO:teuthology.orchestra.run.smithi078.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.31554.log.gz 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28756.log 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30478.log: /var/log/ceph/ceph-client.admin.33110.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30449.log.gz 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30780.log 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.30478.log.gz 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.28756.log: -- /var/log/ceph/ceph-client.admin.31856.log 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33110.log.gz 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30780.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28756.log.gz 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31856.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30622.log 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.30780.log.gz 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30955.log 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.31856.log.gz 2024-09-15T20:51:06.727 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30622.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31986.log 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30955.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32952.log 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30622.log.gz 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31986.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32687.log 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30955.log.gz 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29068.log 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32952.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.31986.log.gz 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33359.log 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32687.log: 88.6%/var/log/ceph/ceph-client.admin.29068.log: -- replaced with /var/log/ceph/ceph-client.admin.32952.log.gz 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32300.log 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32687.log.gz 2024-09-15T20:51:06.732 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33359.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.29068.log.gz 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33359.log.gz 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33004.log 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32300.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30345.log 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33004.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33645.log 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32300.log.gz 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30345.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25125.log 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33004.log.gz 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.30345.log.gz 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33645.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33645.log.gz 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32196.log 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.25125.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30648.log 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32196.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29094.log 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.25125.log.gz 2024-09-15T20:51:06.733 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30648.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31636.log 2024-09-15T20:51:06.734 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32196.log.gz 2024-09-15T20:51:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28802.log 2024-09-15T20:51:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29491.log 2024-09-15T20:51:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28802.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28971.log 2024-09-15T20:51:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29491.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28802.log.gz 2024-09-15T20:51:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28193.log 2024-09-15T20:51:06.744 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28794.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28795.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28193.log: /var/log/ceph/ceph-client.admin.28794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29804.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29175.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr: 91.2%/var/log/ceph/ceph-client.admin.29804.log: -- replaced with /var/log/ceph/ceph-client.admin.29491.log.gz 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29490.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28795.log.gz 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:gzip/var/log/ceph/ceph-client.admin.29175.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.28169.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29490.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28652.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.28169.log: /var/log/ceph/ceph-client.admin.25248.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr: 90.8%/var/log/ceph/ceph-client.admin.28652.log: -- replaced with /var/log/ceph/ceph-client.admin.28794.log.gz 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29804.log.gz 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29385.log 2024-09-15T20:51:06.745 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28633.log 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.25248.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.29490.log.gz 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28584.log 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28633.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.25248.log.gz 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28320.log 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28584.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29385.log.gz 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28502.log 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.28633.log.gz 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28320.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29050.log 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr: 93.8% -- replaced with /var/log/ceph/ceph-client.admin.28652.log.gz 2024-09-15T20:51:06.746 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29066.log 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32765.log 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.30648.log.gz 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29094.log: /var/log/ceph/ceph-client.admin.31636.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33192.log 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.29094.log.gz/var/log/ceph/ceph-client.admin.32765.log: 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31882.log 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.31636.log.gz 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33903.log 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33192.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.32765.log.gz 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31882.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31340.log 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.33192.log.gz 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30754.log 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31882.log.gz 2024-09-15T20:51:06.754 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33903.log: /var/log/ceph/ceph-client.admin.31340.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31446.log 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.33903.log.gz 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31340.log.gz 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30423.log 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31446.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31087.log 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30754.log.gz 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30423.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28062.log 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31446.log.gz 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33385.log 87.8% 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.30423.log.gz 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31960.log 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.31087.log.gz/var/log/ceph/ceph-client.admin.28062.log: 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30163.log 2024-09-15T20:51:06.755 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4%gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.28062.log.gz 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3%/var/log/ceph/ceph-client.admin.31960.log: -- replaced with /var/log/ceph/ceph-client.admin.33385.log.gz 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30163.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.31960.log.gz 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.30163.log.gz 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31113.log 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28730.log 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33248.log 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28251.log 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.31113.log.gz 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33248.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28730.log.gz 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33248.log.gz 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31472.log 2024-09-15T20:51:06.756 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28251.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33879.log 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31472.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28556.log 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.28251.log.gz 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33879.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.31472.log.gz 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33619.log 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.33879.log.gz 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28556.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21527.log 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28552.log 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.28556.log.gz 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.tmp-client.admin.21527.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21527.log.gz 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33619.log.gz 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28249.log 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28552.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31686.log 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28249.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28886.log 2024-09-15T20:51:06.757 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28552.log.gz 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31686.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28249.log.gz 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.31686.log.gz 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28452.log 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28886.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32818.log 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28452.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28886.log.gz 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32818.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.28452.log.gz 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32818.log.gz 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33222.log 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32433.log 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.33222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32926.log 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32433.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.33222.log.gz 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.32433.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28449.log 2024-09-15T20:51:06.758 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:06.759 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32926.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31751.log 2024-09-15T20:51:06.759 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28449.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32926.log.gz 2024-09-15T20:51:06.759 INFO:teuthology.orchestra.run.smithi078.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.28449.log.gz 2024-09-15T20:51:06.759 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32583.log 2024-09-15T20:51:06.759 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33489.log 2024-09-15T20:51:06.759 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32583.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.31751.log.gz 2024-09-15T20:51:06.769 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29050.log: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.28584.log.gz 2024-09-15T20:51:06.769 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28177.log 2024-09-15T20:51:06.769 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29066.log: 93.6% -- replaced with /var/log/ceph/ceph-client.admin.28193.log.gz 2024-09-15T20:51:06.769 INFO:teuthology.orchestra.run.smithi146.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.28502.log.gz 2024-09-15T20:51:06.769 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29436.log 2024-09-15T20:51:06.769 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28177.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29489.log 2024-09-15T20:51:06.769 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29436.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.29066.log.gz 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29387.log 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.29050.log.gz 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29489.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29387.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29436.log.gz 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29018.log 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.3.log: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.29387.log.gz 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29489.log.gz 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29488.log 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29018.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29488.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.29488.log.gz 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29438.log 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-osd.5.log: 92.6% -- replaced with /var/log/ceph/ceph-client.admin.28971.log.gz 2024-09-15T20:51:06.770 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28217.log 2024-09-15T20:51:06.771 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29438.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28169.log.gz 2024-09-15T20:51:06.771 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29802.log 2024-09-15T20:51:06.771 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28217.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29438.log.gz 2024-09-15T20:51:06.771 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28857.log 2024-09-15T20:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29802.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28217.log.gz 2024-09-15T20:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29495.log 2024-09-15T20:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28857.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29802.log.gz 2024-09-15T20:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29157.log 2024-09-15T20:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr: 86.8%/var/log/ceph/ceph-client.admin.29495.log: -- replaced with /var/log/ceph/ceph-client.admin.28857.log.gz 2024-09-15T20:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25296.log 2024-09-15T20:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29157.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.29157.log.gz 2024-09-15T20:51:06.777 INFO:teuthology.orchestra.run.smithi146.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29495.log.gz 2024-09-15T20:51:06.778 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29494.log 2024-09-15T20:51:06.778 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.25296.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28435.log 2024-09-15T20:51:06.778 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29494.log: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.25296.log.gz 2024-09-15T20:51:06.778 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25224.log 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28435.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29494.log.gz 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28423.log 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.25224.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25224.log.gz 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29493.log 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.28435.log.gz/var/log/ceph/ceph-client.admin.28423.log: 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29496.log 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29493.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.29493.log.gz 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28382.log 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.29018.log.gz 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr: 92.7% -- replaced with /var/log/ceph/ceph-client.admin.29175.log.gz 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28644.log 2024-09-15T20:51:06.786 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29496.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.28423.log.gz 2024-09-15T20:51:06.787 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28382.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29133.log 2024-09-15T20:51:06.787 INFO:teuthology.orchestra.run.smithi146.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.29496.log.gz/var/log/ceph/ceph-client.admin.28644.log: 2024-09-15T20:51:06.787 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28225.log 2024-09-15T20:51:06.787 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28616.log 2024-09-15T20:51:06.794 INFO:teuthology.orchestra.run.smithi078.stderr: 89.1%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31392.log 2024-09-15T20:51:06.794 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.32583.log.gz 2024-09-15T20:51:06.794 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33489.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30570.log 2024-09-15T20:51:06.794 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31392.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33489.log.gz 2024-09-15T20:51:06.794 INFO:teuthology.orchestra.run.smithi078.stderr: 91.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T20:51:06.794 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- 88.3% /var/log/ceph/ceph-client.admin.29821.log -- replaced with /var/log/ceph/ceph-client.admin.31392.log.gz 2024-09-15T20:51:06.794 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32713.log 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30069.log 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.30570.log.gz 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr: 86.2%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29847.log 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.29821.log.gz 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32713.log: /var/log/ceph/ceph-client.admin.30069.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31908.log 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29847.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32713.log.gz 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.30069.log.gz 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32326.log 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29847.log.gz/var/log/ceph/ceph-client.admin.31908.log: 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33030.log 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T20:51:06.795 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32326.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.31908.log.gz 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33030.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28184.log 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph.audit.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.32326.log.gz 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33030.log.gz 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29042.log 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28248.log 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28184.log: /var/log/ceph/ceph-client.admin.29042.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28184.log.gz 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.29042.log.gzgzip 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.32012.log 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28248.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29016.log 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.28248.log.gz 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32012.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28581.log 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4%/var/log/ceph/ceph-client.admin.29016.log: -- replaced with /var/log/ceph/ceph-client.admin.32012.log.gz 2024-09-15T20:51:06.796 INFO:teuthology.orchestra.run.smithi078.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.29016.log.gz 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33058.log 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33411.log 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28581.log: /var/log/ceph/ceph-client.admin.33058.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.28581.log.gz 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.33058.log.gz 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31314.log 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30267.log 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31314.log: /var/log/ceph/ceph-client.admin.33411.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33411.log.gz 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31314.log.gz 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33853.log 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30702.log 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30267.log: /var/log/ceph/ceph-client.admin.33853.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.30267.log.gz 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33853.log.gz 2024-09-15T20:51:06.797 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28782.log 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29656.log 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30702.log: /var/log/ceph/ceph-client.admin.28782.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.30702.log.gz 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31420.log 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28782.log.gz 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29656.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.29656.log.gz 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31248.log 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33723.log 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31420.log: /var/log/ceph/ceph-client.admin.31248.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.31420.log.gz 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr: 86.5%gzip -- replaced with /var/log/ceph/ceph-client.admin.31248.log.gz 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.28618.log 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr: 93.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T20:51:06.798 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33723.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.33723.log.gz 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28704.log 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28618.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32222.log 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28990.log 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28618.log.gz 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32222.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.28704.log.gz 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.32222.log.gz 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33593.log 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31803.log 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28990.log: /var/log/ceph/ceph-client.admin.33593.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33307.log 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31803.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.28990.log.gz 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.33593.log.gz 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:gzip 88.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.31803.log.gz 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr: --verbose -- /var/log/ceph/ceph-client.admin.29795.log 2024-09-15T20:51:06.799 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30877.log 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33307.log: /var/log/ceph/ceph-client.admin.29795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32609.log 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30877.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.33307.log.gz 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.29795.log.gz 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.30877.log.gz 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32404.log 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32609.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31498.log 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32404.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33697.log 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32609.log.gz 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31498.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.32404.log.gz 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.31498.log.gz 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31165.log 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25317.log 2024-09-15T20:51:06.800 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33697.log: /var/log/ceph/ceph-client.admin.31165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29685.log 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.25317.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.33697.log.gz 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.31165.log.gz 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.25317.log.gz 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28391.log 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28808.log/var/log/ceph/ceph-client.admin.29685.log: 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28391.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33463.log 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.29685.log.gz 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3%/var/log/ceph/ceph-client.admin.28808.log: -- replaced with /var/log/ceph/ceph-client.admin.28391.log.gz 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.28808.log.gz 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32635.log 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32557.log 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33463.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30215.log 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32635.log: /var/log/ceph/ceph-client.admin.32557.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.33463.log.gz 2024-09-15T20:51:06.801 INFO:teuthology.orchestra.run.smithi078.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.32635.log.gz 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32557.log.gz 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30981.log 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30215.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28548.log 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30981.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33749.log 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.30215.log.gz 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28548.log: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.30981.log.gz 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.28548.log.gz 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.31934.log 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29903.log 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33749.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32170.log 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.31934.log: /var/log/ceph/ceph-client.admin.29903.log: 88.4% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.31934.log.gz 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr: -- replaced with /var/log/ceph/ceph-client.admin.33749.log.gz 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.29903.log.gz 2024-09-15T20:51:06.802 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.33166.log 2024-09-15T20:51:06.803 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32274.log 2024-09-15T20:51:06.803 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.33166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28912.log 2024-09-15T20:51:06.803 INFO:teuthology.orchestra.run.smithi078.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.32170.log.gz 2024-09-15T20:51:06.803 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32274.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.33166.log.gz 2024-09-15T20:51:06.803 INFO:teuthology.orchestra.run.smithi078.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.32274.log.gz 2024-09-15T20:51:06.809 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28225.log: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.28225.log.gz 2024-09-15T20:51:06.809 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29386.log 2024-09-15T20:51:06.809 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28616.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.28382.log.gz 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29386.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29386.log.gz 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28604.log 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.29133.log.gz 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28604.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.28616.log.gz 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28792.log 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.log: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.28604.log.gz 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25272.log 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28792.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.28177.log.gz 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28475.log 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.25272.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25272.log.gz 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28877.log 2024-09-15T20:51:06.810 INFO:teuthology.orchestra.run.smithi146.stderr: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.28792.log.gz 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28475.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29803.log 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28877.log: 86.9% -- replaced with /var/log/ceph/ceph.log.gz 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28185.log 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30397.log 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32352.log 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.28912.log: /var/log/ceph/ceph-client.admin.30397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29120.log 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr: 87.3%/var/log/ceph/ceph-client.admin.32352.log: -- replaced with /var/log/ceph/ceph-client.admin.28912.log.gz 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.30397.log.gz 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.32352.log.gz 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.32739.log 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29120.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29763.log 2024-09-15T20:51:06.811 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.32739.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.30371.log 2024-09-15T20:51:06.812 INFO:teuthology.orchestra.run.smithi078.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.29120.log.gz 2024-09-15T20:51:06.812 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.29763.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.32739.log.gz 2024-09-15T20:51:06.812 INFO:teuthology.orchestra.run.smithi078.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.29763.log.gz 2024-09-15T20:51:06.812 INFO:teuthology.orchestra.run.smithi078.stderr:/var/log/ceph/ceph-client.admin.30371.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.30371.log.gz 2024-09-15T20:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29803.log: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.28475.log.gz 2024-09-15T20:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25200.log 2024-09-15T20:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28185.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.29803.log.gz 2024-09-15T20:51:06.813 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28201.log 2024-09-15T20:51:06.814 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.25200.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.25200.log.gz 2024-09-15T20:51:06.814 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-09-15T20:51:06.814 INFO:teuthology.orchestra.run.smithi146.stderr: 90.6% -- replaced with /var/log/ceph/ceph-client.admin.28877.log.gz 2024-09-15T20:51:06.814 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28201.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28280.log 2024-09-15T20:51:06.815 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph.audit.log: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.28201.log.gz 2024-09-15T20:51:06.816 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28413.log 2024-09-15T20:51:06.816 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28280.log: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.28185.log.gz 2024-09-15T20:51:06.817 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28782.log 2024-09-15T20:51:06.823 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28413.log: 92.3% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-09-15T20:51:06.826 INFO:teuthology.orchestra.run.smithi146.stderr: 91.4% -- replaced with /var/log/ceph/ceph-client.admin.28413.log.gz 2024-09-15T20:51:06.831 INFO:teuthology.orchestra.run.smithi146.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.28644.log.gz 2024-09-15T20:51:06.835 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28936.log 2024-09-15T20:51:06.835 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.25176.log 2024-09-15T20:51:06.836 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28936.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29487.log 2024-09-15T20:51:06.836 INFO:teuthology.orchestra.run.smithi146.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.28782.log.gz 2024-09-15T20:51:06.837 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.25176.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.25176.log.gz 2024-09-15T20:51:06.837 INFO:teuthology.orchestra.run.smithi146.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.28936.log.gz 2024-09-15T20:51:06.837 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-09-15T20:51:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29487.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29437.log 2024-09-15T20:51:06.838 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28785.log 2024-09-15T20:51:06.840 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29437.log: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.29487.log.gz 2024-09-15T20:51:06.840 INFO:teuthology.orchestra.run.smithi146.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.29437.log.gz 2024-09-15T20:51:06.841 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-09-15T20:51:06.841 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28745.log 2024-09-15T20:51:06.846 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28785.log: /var/log/ceph/ceph-osd.4.log: 92.2% -- replaced with /var/log/ceph/ceph-client.admin.28785.log.gz 2024-09-15T20:51:06.846 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28856.log 2024-09-15T20:51:06.847 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28745.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.28745.log.gz 2024-09-15T20:51:06.848 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28793.log 2024-09-15T20:51:06.850 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28856.log: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.28856.log.gz 2024-09-15T20:51:06.851 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28209.log 2024-09-15T20:51:06.853 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28793.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28793.log.gz 2024-09-15T20:51:06.854 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29120.log 2024-09-15T20:51:06.871 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28209.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28230.log 2024-09-15T20:51:06.882 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29120.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.29120.log.gz 2024-09-15T20:51:06.883 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.29492.log 2024-09-15T20:51:06.898 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28230.log: 95.0% -- replaced with /var/log/ceph/ceph-client.admin.28280.log.gz 2024-09-15T20:51:06.899 INFO:teuthology.orchestra.run.smithi146.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.28339.log 2024-09-15T20:51:06.902 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.29492.log: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.29492.log.gz 2024-09-15T20:51:06.927 INFO:teuthology.orchestra.run.smithi146.stderr:/var/log/ceph/ceph-client.admin.28339.log: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28339.log.gz 2024-09-15T20:51:06.928 INFO:teuthology.orchestra.run.smithi146.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.28209.log.gz 2024-09-15T20:51:07.095 INFO:teuthology.orchestra.run.smithi146.stderr: 91.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-09-15T20:51:07.259 INFO:teuthology.orchestra.run.smithi146.stderr: 93.2% -- replaced with /var/log/ceph/ceph-client.admin.28320.log.gz 2024-09-15T20:51:07.358 INFO:teuthology.orchestra.run.smithi078.stderr: 90.3% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-09-15T20:51:07.732 INFO:teuthology.orchestra.run.smithi146.stderr: 91.2% -- replaced with /var/log/ceph/ceph-client.admin.28230.log.gz 2024-09-15T20:51:08.522 INFO:teuthology.orchestra.run.smithi078.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-09-15T20:51:08.525 INFO:teuthology.orchestra.run.smithi146.stderr: 91.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-09-15T20:51:13.712 INFO:teuthology.orchestra.run.smithi146.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-09-15T20:51:13.741 INFO:teuthology.orchestra.run.smithi078.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-09-15T20:51:13.966 INFO:teuthology.orchestra.run.smithi078.stderr: 92.4% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-09-15T20:51:14.029 INFO:teuthology.orchestra.run.smithi146.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-09-15T20:51:14.098 INFO:teuthology.orchestra.run.smithi146.stderr: 92.6% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-09-15T20:51:14.100 INFO:teuthology.orchestra.run.smithi146.stderr: 2024-09-15T20:51:14.100 INFO:teuthology.orchestra.run.smithi146.stderr:real 0m7.386s 2024-09-15T20:51:14.100 INFO:teuthology.orchestra.run.smithi146.stderr:user 0m24.658s 2024-09-15T20:51:14.100 INFO:teuthology.orchestra.run.smithi146.stderr:sys 0m0.836s 2024-09-15T20:51:14.180 INFO:teuthology.orchestra.run.smithi078.stderr: 92.5% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-09-15T20:51:14.181 INFO:teuthology.orchestra.run.smithi078.stderr: 2024-09-15T20:51:14.182 INFO:teuthology.orchestra.run.smithi078.stderr:real 0m7.508s 2024-09-15T20:51:14.182 INFO:teuthology.orchestra.run.smithi078.stderr:user 0m23.707s 2024-09-15T20:51:14.182 INFO:teuthology.orchestra.run.smithi078.stderr:sys 0m0.811s 2024-09-15T20:51:14.183 INFO:tasks.ceph:Archiving logs... 2024-09-15T20:51:14.183 DEBUG:teuthology.misc:Transferring archived files from smithi078:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/remote/smithi078/log 2024-09-15T20:51:14.184 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T20:51:17.603 DEBUG:teuthology.misc:Transferring archived files from smithi146:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/remote/smithi146/log 2024-09-15T20:51:17.604 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-15T20:51:20.930 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-15T20:51:20.941 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-15T20:51:20.942 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T20:51:20.944 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-15T20:51:21.014 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-15T20:51:21.023 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-15T20:51:21.023 DEBUG:teuthology.orchestra.run.smithi078:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:51:21.026 DEBUG:teuthology.orchestra.run.smithi146:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-15T20:51:21.036 INFO:teuthology.orchestra.run.smithi078.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T20:51:21.036 INFO:teuthology.orchestra.run.smithi078.stdout:============================================================================== 2024-09-15T20:51:21.036 INFO:teuthology.orchestra.run.smithi078.stdout:+hv01.front.sepi 67.205.162.81 3 u 31 64 377 0.092 -3.537 0.943 2024-09-15T20:51:21.036 INFO:teuthology.orchestra.run.smithi078.stdout:*hv02.front.sepi 63.231.80.2 3 u 34 64 377 0.098 -3.884 0.447 2024-09-15T20:51:21.036 INFO:teuthology.orchestra.run.smithi078.stdout:+hv03.front.sepi 74.6.168.72 3 u 22 64 377 0.088 -2.597 0.411 2024-09-15T20:51:21.036 INFO:teuthology.orchestra.run.smithi078.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-15T20:51:21.038 INFO:teuthology.orchestra.run.smithi146.stdout: remote refid st t when poll reach delay offset jitter 2024-09-15T20:51:21.038 INFO:teuthology.orchestra.run.smithi146.stdout:============================================================================== 2024-09-15T20:51:21.038 INFO:teuthology.orchestra.run.smithi146.stdout:+hv01.front.sepi 67.205.162.81 3 u 31 64 377 0.072 -3.655 0.548 2024-09-15T20:51:21.038 INFO:teuthology.orchestra.run.smithi146.stdout:*hv02.front.sepi 63.231.80.2 3 u 30 64 377 0.066 -2.960 0.421 2024-09-15T20:51:21.039 INFO:teuthology.orchestra.run.smithi146.stdout:+hv03.front.sepi 74.6.168.72 3 u 26 64 377 0.093 -1.708 0.510 2024-09-15T20:51:21.039 INFO:teuthology.orchestra.run.smithi146.stdout: hv04.front.sepi .INIT. 16 u - 512 0 0.000 +0.000 0.000 2024-09-15T20:51:21.039 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-15T20:51:21.050 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-15T20:51:21.051 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-15T20:51:21.069 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-15T20:51:21.080 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-15T20:51:21.095 INFO:teuthology.task.internal:Duration was 1996.438868 seconds 2024-09-15T20:51:21.096 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-15T20:51:21.111 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-15T20:51:21.111 DEBUG:teuthology.orchestra.run.smithi078:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T20:51:21.114 DEBUG:teuthology.orchestra.run.smithi146:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-15T20:51:21.154 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-15T20:51:21.154 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi078.front.sepia.ceph.com 2024-09-15T20:51:21.155 DEBUG:teuthology.orchestra.run.smithi078:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T20:51:21.203 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi146.front.sepia.ceph.com 2024-09-15T20:51:21.204 DEBUG:teuthology.orchestra.run.smithi146:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-15T20:51:21.213 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-15T20:51:21.213 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T20:51:21.248 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-15T20:51:21.285 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-15T20:51:21.285 DEBUG:teuthology.orchestra.run.smithi078:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T20:51:21.316 DEBUG:teuthology.orchestra.run.smithi146:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-15T20:51:21.389 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-15T20:51:21.402 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-15T20:51:21.402 DEBUG:teuthology.orchestra.run.smithi078:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T20:51:21.413 DEBUG:teuthology.orchestra.run.smithi146:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-15T20:51:21.425 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-15T20:51:21.435 DEBUG:teuthology.orchestra.run.smithi078:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:51:21.460 DEBUG:teuthology.orchestra.run.smithi146:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:51:21.469 INFO:teuthology.orchestra.run.smithi078.stdout:kernel.core_pattern = core 2024-09-15T20:51:21.475 INFO:teuthology.orchestra.run.smithi146.stdout:kernel.core_pattern = core 2024-09-15T20:51:21.491 DEBUG:teuthology.orchestra.run.smithi078:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:51:21.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:51:21.539 DEBUG:teuthology.orchestra.run.smithi146:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-15T20:51:21.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-15T20:51:21.544 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-15T20:51:21.553 INFO:teuthology.task.internal:Transferring archived files... 2024-09-15T20:51:21.554 DEBUG:teuthology.misc:Transferring archived files from smithi078:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/remote/smithi078 2024-09-15T20:51:21.554 DEBUG:teuthology.orchestra.run.smithi078:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T20:51:21.608 DEBUG:teuthology.misc:Transferring archived files from smithi146:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-15_20:00:16-rados-main-distro-default-smithi/7905600/remote/smithi146 2024-09-15T20:51:21.608 DEBUG:teuthology.orchestra.run.smithi146:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-15T20:51:21.653 INFO:teuthology.task.internal:Removing archive directory... 2024-09-15T20:51:21.653 DEBUG:teuthology.orchestra.run.smithi078:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T20:51:21.656 DEBUG:teuthology.orchestra.run.smithi146:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-15T20:51:21.680 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-15T20:51:21.694 INFO:teuthology.task.internal:Not uploading archives. 2024-09-15T20:51:21.695 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-15T20:51:21.710 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-15T20:51:21.710 DEBUG:teuthology.orchestra.run.smithi078:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T20:51:21.712 DEBUG:teuthology.orchestra.run.smithi146:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-15T20:51:21.715 INFO:teuthology.orchestra.run.smithi078.stdout: 658465 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 20:51 /home/ubuntu/cephtest 2024-09-15T20:51:21.727 INFO:teuthology.orchestra.run.smithi146.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 15 20:51 /home/ubuntu/cephtest 2024-09-15T20:51:21.729 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-15T20:51:21.739 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-15T20:51:21.766 INFO:teuthology.run:Summary data: description: rados/monthrash/{ceph clusters/3-mons mon_election/classic msgr-failures/few msgr/async-v1only objectstore/{bluestore-options/write$/{write_v2} bluestore/bluestore-comp-snappy} rados supported-random-distro$/{ubuntu_latest} thrashers/sync workloads/rados_api_tests} duration: 1996.4388680458069 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-15T20:51:21.766 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-15T20:51:21.859 INFO:teuthology.run:pass